SAIN: Search-And-INfer, A Mathematical and Computational Framework for Personalised Multimodal Data Modelling with Applications in Health Care

1. Introduction

Multimodal data has been gathered at a personalised level in large quantities, worldwide, for many applications, such as neuro-imaging analysis, personalised health diagnosis and prognosis, environmental modelling, and financial modelling, to mention only a few of them. Still, there are no efficient methods to integrate various multimodal data for a new subject and derive a more accurate and explainable diagnosis or prognosis based on existing multimodal data of many other subjects. The goal of this paper is to create such a method.

There are three main approaches to multimodal data integration in machine learning, explored so far [1]:

1. Early integration, where a common vector represents all modalities used for training a model and for its recall. This approach has been used for integrating time series and textual information [2,3], for image integration [4], and for the integration of clinical, social, and cognitive data modalities to predict psychosis in young adults [5]. While the method in [4] is based on deep, feedforward neural networks, the methods in [2,5] use a brain-inspired spiking neural network architecture NeuCube [6].

2. Late integration, where a model is created and trained for each of the modalities of data, and the results from all models are integrated to calculate the output. This approach has been demonstrated in [1] on integrating clinical, genetic, cognitive, and social data for medical prognosis.

3. Hybrid, early, late, and intermediate integration of data modalities, where the two approaches above are combined [1].

The proposed SAIN method in this paper is designed for early integration of data modalities, where specific encoding and distance metrics are suggested for different types of data, along with novel algorithms for search in a multimodal database and inference. These search and inference algorithms are related to building a personalised model for individual outcome assessment and its explanation.

Personalised modelling is concerned with the creation of an individual model for a new personal (individual) record of data X, using an already existing repository D of many other personal records for which the outcomes are known, to assess the outcome of the new record X [7]. Methods for personalised modelling have been developed to work mainly on a single modality of data [8,9]. These methods have been applied in many applications and constitute the state-of-the-art in the field (e.g. [1,9,10,11,12,13]). In [9,14], a personalised modelling is proposed based on static and temporal modalities of data, which are used to train a spiking neural network model.

The enormous growth of personal multimodal data worldwide demands more advanced methods for personal modelling with the use of multimodal data. This paper offers such a method, called SAIN, where the specificity of each data modality is considered and new algorithms are proposed for the encoding of multimodal data, for search in a multimodal data repository, and for multimodal inference, along with its explanation and visualisation. In contrast with the statistical solutions used in [15,16,17], we adopt a probabilistic framework that gives more precise evaluations of probabilities of outcomes for an individual.

2. Mathematical Description

In this section, we present the mathematical method. In detail, we start with the database coding and a class of distances (metrics) to be used in this article, followed by the list of tasks (problems) and their solutions. Detailed examples illustrate the solutions. Detailed solutions to three critical problems, survival analysis, heart disease diagnosis, and time series classification, are presented in detail and illustrated with numerical examples.

2.1. Database

We will work with the multidimensional data described as follows:

• $m>1$ objects (samples) $o_1,\dots ,o_m$,
• each object $o_i$ ($1\le i\le m$) is defined by $n>1$ criteria (variables) $c_1,\dots ,c_n$ with values in linearly ordered domains $D_i$ with $min{D}_i$ and $max{D}_i$; if some value $a_{i,j}\in D_i$ ($1\le i\le m$, $1\le j\le n$) is either missing or uncertain, then its value is recorded as ∞,
• $n>1$ weights $w_1,\dots ,w_n$ in $(0,1)$ with ${\sum }_{i=1}^n{w}_i=1$, where each $w_i$ ($1\le i\le n$) quantifies the importance of the criterion $c_i$; if $w_i={\displaystyle \frac{1}{n}}$ for all $1\le i\le n$, then all criteria are equally important; a criterion $c_i$ is ignore if $w_i=0$.

Data of independent variables are organised as in (Table 1):

2.2. Distance Metrics

A distance metric on a space X is a positive real-valued function $d:X\times X\to {\mathbf{R}}_{+}$ satisfying the following three conditions for all $x,y,z\in X$: a) $d(x,y)=0$ if and only if $x=y$, b) $d(x,y)=d(y,x)$, c) $d(x,z)\le d(x,y)+d(y,z)$.

The multicriteria metrics [18,19] (used in multicriteria recommendation systems [20]) presented in this part can be used on a variety of domains $X=D_i$: they can be sets of logical values, rational numbers, percentages, digitally codified images, sounds, videos, and many others. We use a bounded distributive complemented lattice $(L,\vee ,\wedge ,\overline{\phantom{x}},0,1)$ to describe uniformly the domains $D_i$. We rank all objects according to their aggregated distance to a new one; based on that, we calculate probabilities of the new object belonging to different classes, represented in the object repository.

Here is a list with illustrative, but far from exhaustive, examples of domains $D_i$:

• Logical Boolean domain: $(\{0,1\},max,min,\overline{\phantom{x}},0,1)$, where $\overline{x}=1-x,x\in \{0,1\}$.
• Logical non-Boolean domain: $\left(\left\{0,{\displaystyle \frac{1}{N-1}},{\displaystyle \frac{2}{N-1}},\dots ,{\displaystyle \frac{N-2}{N-1}},1\right\},max,min,\overline{\phantom{x}},0,1\right)$, where $x\in \left\{0,{\displaystyle \frac{1}{N-1}},{\displaystyle \frac{2}{N-1}},\dots ,{\displaystyle \frac{N-2}{N-1}},1\right\}$ and $\overline{x}=1-x,$.
• Numerical domain with natural values: $\left(\{0,1,\dots ,N\},max,min,\overline{\phantom{x}},0,1\right)$, where $\overline{x}=N-x,x\in \{0,1,\dots ,N\}$.
• Numerical domain with rational values: $\left(\{x\mid a\le x\le A\},max,min,\overline{\phantom{x}},a,A\right)$, where $\overline{x}=A-x,a\le x\le A$.
• Binary code: $\left({\{0,1\}}^n,max,min,\overline{\phantom{x}},00\dots 0,11\dots 1\right)$, where the domain consists of all binary strings of length n, ${\{0,1\}}^n=\{x_1{x}_2\dots x_n\mid x_i\in \{0,1\}\}$ and for all $x_1{x}_2\dots x_n,y_1{y}_2\dots y_n\in {\{0,1\}}^n$, $max(x_1{x}_2\dots x_n,y_1{y}_2\dots y_n)=max(x_1,y_1)max(x_2,y_2)\dots max(x_n,y_n)$, $min(x_1{x}_2\dots x_n,y_1{y}_2\dots y_n)=min(x_1,y_1)min(x_2,y_2)\dots min(x_n,y_n)$, $\overline{x_1{x}_2\dots x_n}=(1-x_1)(1-x_2)\dots (1-x_n).$

In the lattice $(L,\vee ,\wedge ,\overline{\phantom{x}},0,1)$ we introduce, following [18], the metric:

$$d(x,y)=\left\{\begin{array}{cc}(x\wedge \overline{y})\vee (\overline{x}\wedge y),\hfill & \mathrm{if}x\ne y,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

for $x,y\in L$. This metric d can be extended to $L\cup \{\infty \}$ as follows:

$$d_{\infty }(x,y)=\left\{\begin{array}{cc}d(x,y),\hfill & \mathrm{if}x,y\in L,\hfill \\ \sigma \left(x\right),\hfill & \mathrm{if}x\in L\mathrm{and}y=\infty ,\hfill \\ \sigma \left(y\right),\hfill & \mathrm{if}y\in L\mathrm{and}x=\infty ,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

where $\sigma \left(x\right)=max(x,\overline{x})$.

The metrics $d_{\infty ,i}$ on $L_i\cup \{\infty \}$, $1\le i\le n$, can be extended to ${(L_i\cup \{\infty \})}^n$, i.e. to n-dimensional vectors, as follows:

$$d_{\infty }(x_1{x}_2\dots x_n,y_1{y}_2\dots y_n)=\sum _{i=1}^n{d}_{\infty ,i}(x_i,y_i),$$

(1)

where $x_i,y_i\in L_i\cup \{\infty \}$, $1\le i\le n$.

In what follows, we write d for $d_{\infty }$ when the meaning is clear from the context.

2.3. Tasks Specification

Data organised as in Table 2, Table 3, and Table 4 consists of independent objects augmented with a column of labels, the weights of criteria, and a new unlabelled object, respectively.

Additional information associated with data in Table 2 may include the range of each criterion $c_j$ and the associated specific distance, e.g. the Euclidean distance for real numbers and the distance d for binary strings or strings over a non-binary alphabet (e.g. for images or colours).

We consider the following tasks:

Task 1:

Calculate the distance (or similarity metric) between the new object and each object in Table 2.

If the distance corresponding to $c_i$ is $d_i$, then

$$d(o_j,x)=\sum _{i=1}^n{w}_i·d_i(a_{i,j},x_j).$$

Task 2:

Given a threshold $\delta >0$, calculate all objects $o_i$ at a distance at most $\delta$ to x.

Task 3:

Calculate the probability of a new object belonging to a labelled class (e.g. low risk vs. high risk) using a threshold $\delta$ and Table 2.

Task 4:

Rank the criteria in Table 2 and calculate the marker or markers criterion/criteria that are the most important/ones.

Task 5:

Assign alternative weights to criteria.

Task 6:

Test the data accuracy and method for Task 4.

2.4. Tasks Solutions

For Task 1 we calculate the distances $d_{\infty }(o_i,x)$ between each object $o_i$ in Table 2 and x in Table 4.

For Task 2, given a threshold $\delta >0$, we calculate all objects in Table 2 at a distance at most $\delta$ to x, that is, the objects which are $\delta$-similar to x:

$$C_{\delta ,x}=\{o_i\mid d(x,o_i)\le \delta ,1\le i\le m\},$$

and its complement $\overline{C_{\delta ,x}}$.

For Task 3 we calculate the probability that x is in class label $l_t$, which is the ratio of the number of objects in $C_{\delta ,x}$ with the label $l_t$ to the size of the cluster $C_{\delta ,x}$:

$$Prob\left(x\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}{l}_t\right)={\displaystyle \frac{\#\{o_i\in C_{\delta ,x}\mid l_i=l_t\}}{\#\left(C_{\delta ,x}\right)}},$$

where $\#\{\dots \}$ means the number of elements in the set $\{\dots \}$.

For Task 4, we work with Table 2. Recall that for each criterion $c_i$ we have a domain $D_i$ augmented with information “high" or “low," indicating whether higher or lower values are desirable. Based on this information, we can construct a hypothetical object (see Table 5) which has the most desirable values for each criterion: one could see this object as an “exemplar" one.

Sometimes, criteria are interrelated or correlated. This means that in some cases, there is no unique “exemplar object", but a couple of them have to be studied in ranking the importance of criteria.

For example, fix an “exemplar object" $o_E$.

• Compute the distances $d_{\infty }(o_i,o_E)$ between each object $o_i$ in Table 2 and $o_E$, so obtain a vector with n non-negative real components $V_0=(d_1^0,\dots d_n^0)$.
• For each $1\le t\le m$, compute the distances $d_{\infty }(o_i,o_E)$ taking into consideration all criteria in Table 2 except$c_t$: obtain the vector $V_t=(d_1^t,\dots d_n^t)$.
• Compute the distances between $dist(V_0,V_t)$, $1\le t\le m$ using the formula

  $$dist(V_0,V_t)=\sum _{i=1}^n|d_{0,i}-d_{t,i}|,$$

  and sort them in increasing order. The criterion $c_t$ is a marker if $dist(V_0,V_t)\ge dist(V_0,V_j)$, for every $1\le j\le m$.

We repeat this procedure for each “exemplar object" and study possible variations.

For Task 5, normalise the distances $dist(V_0,V_t)$ and use these values to construct the weights $w_i^{*}$, $1\le t\le m$.

For Task 6 assume we have weights $\left(w_i\right)$ associated to Table 2 (see Table 1). To test the accuracy of the data and method used for Task 4, compare the original weights $\left(w_i\right)$ with $\left(w_i^{*}\right)$. Serious discrepancies should signal issues either with the data or the choices made in the applications of the method.

2.5. An Example

We illustrate the above tasks with an example of a labelled database in (see Table 6) and a new object (see Table 7), all having the following seven characteristics (the last column has the label classes 1 and 2):

$c_1$: real number $\{0-100\}$, e.g. age, weight, BMI etc.;

$c_2$: Boolean value $\{0,1\}$, e.g. gender;

$c_3$: integer number $\{0-10,000\}$, e.g. gene expression;

$c_4$: categorical {small, med, large }, e.g. size of tumour, body size, keywords;

$c_5$: colour {red, yellow, white, black}, e.g. colour of a spot on the body, on the heart;

$c_6$: spike sequence of $\{-1,0,1\}$ e.g. encoded EEG, ECG;

$c_7$: black and white image, e.g. MRI, face image.

In this fictitious example, for simplicity, we didn’t use weights.

The first step is to code the data in Table 6 and Table 7. The new data is in Table 8 and Table 9.

Then, we normalise the data in Table 8 and Table 9 – the entries in the first, third, and fourth columns have been divided by 100, 10,000, and 2, respectively. The entries in the last three columns have been transformed into reals in the unit interval, and the column of labels has been removed. In this way, we have obtained Table 10 and Table 11.

Then, we choose an appropriate distance according to each criterion. In this example, we used the Euclidean distance for all criteria (see Table 12 and Table 13).

We can compute $C_{\delta ,x}=\{o_i\mid d(o_i,x)\le \delta \}$ and, accordingly, the probability that x would be labelled in class 1 or class 2.

If $\delta =3.5$, then $C_{3.5,x}=\{o_1,o_2,o_3,o_5,o_6,o_7,o_8\}$ so the probability that x is in class 1 is 2/7 and the probability that x is in class 2 is 5/7. If $\delta =2.5$, then its closest cluster is $C_{2.5,x}=\{o_2,o_3,o_5,o_6,o_7,o_8\}$, so the probability that x is in class 1 is 1/3 and the probability that x is in class 2 is 2/3.

which induces a ranking of the objects in the Table 8: $o_3,o_6,o_7,o_5,o_2,o_8,o_1,o_9,o_2,$.

For Task 4, assume that the criteria $c_1,\dots ,c_7$ in Table 10 have the additional information $(m,m,m,m,m,M,M)$, where m (M) means that the exemplar value is the minimum (maximum) value. Based on this vector, we compute the exemplar object(see Table 14):

Next we calculate $V_0,\dots ,V_t$, ( see Table 15), and finally the distances $Dist(V_0,V_t)$, $t=1,2,\dots ,7$ and the weights as their normalised values, see Table 16. The marker, in this case, is the criterion $c_5$.

2.6. Complexity Estimation of the SAIN Method

The proposed method to compute the similarity between a new object and N objects in a data repository is linear in N, so very fast.

3. Survival Analysis in SAIN

Medical survival analysis evaluates the time until an event of interest occurs, like death or disease recurrence, in a group of patients. This analysis is often used to compare treatment outcomes or predict prognosis. In contrast with the statistical solutions used in [15,16,17], we adopt a probabilistic framework that gives more precise evaluations of probabilities.

3.1. Data and Tasks

We are given the following data:

• Table 17 in which the first column lists the patients treated for the same disease with the same method under strict conditions, and the last column records the times till the patients’ deaths.
• Table 18, which includes the record of the new patient p.
• A threshold $\delta$ which defines the acceptable similarity between p and the relevant $p_i$’s in the Survival database (i.e. $d(p,p_i)\le \delta$).

We consider the following tasks:

Task 1: What is the life expectancy of p?

Task 2: What is the probability that the life expectancy of p is greater than or equal to a given T?

3.2. Tasks Solutions

Using a standard method of survival analysis

• For Task 1,

  (a)

  Compute the set of patients that are similar up to $\delta$ to p:

  $$C_{\delta ,p}=\{p_i\mid d(p,p_i)\le \delta ,1\le i\le m\}.$$

  (2)

  (b)

  Using $C_{\delta ,p}$, compute the probability that p will survive the time $t_j$:

  $$Pro{b}_{\delta }\left(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}{t}_j\right)={\displaystyle \frac{\#\{p_i\in C_{\delta ,p}\mid t_i=t_j\}}{\#\left(C_{\delta ,p}\right)}}.$$

  (3)

  (c)

  Compute the life expectancy of p using the formula:

  $$L{E}_{\delta }\left(p\right)=\sum _{j=1,t_j\in C_{\delta ,p}}^m{t}_j\times Pro{b}_{\delta }\left(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}{t}_j\right).$$

  (4)

• For Task 2, calculate the probability that the life expectancy of p is at least time T:

  $$Pro{b}_{\delta }(LE\left(p\right)\ge T)=\sum _{j=1,t_j\in C_{\delta ,p},t_j\ge T}^{m}Pro{b}_{\delta }\left(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}{t}_j\right).$$

  (5)

3.3. An Example

We illustrate the above tasks with an example of a database in which columns 2–8 record patients’ medical test results, and the last column records time to death (see Table 19) and a new patient (see Table 20):

The distance for column 4 is $d(x,y)=|x-y|=$ and $d_{\infty }(x,\infty )=max(x,1-x)$. For example, $d_{\infty }(1,\infty )=max(1,1-1)=1$. For all other columns, the distance is $d(x,y)=|x-y|$. Finally, the total distance is the sum of individual distances (7 terms), with the results in Table 21.

The results for Task 1, (a), (b), and (c) are listed below:

• For $\delta \ge 3.37$, $C_{\delta ,p}=\{v_1,v_2,v_3,v_4,v_5,v_6,v_7,v_8,v_9\}$, that is the entire database. Then

  (a)

  $L{E}_{\delta }\left(p\right)=50.11$,

  (b)

  i.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=12.3)=1/9$,

  ii.

  $Prob(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=15)=1/9$,

  iii.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=68)=1/9$,

  iv.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=1.4)=1/9$,

  v.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=40.5)=1/9$,

  vi.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=97.2)=2/9$,

  vii.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=55.7)=1/9$,

  viii.

  $Pro{b}_{\delta }(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=63.7)=1/9$.

  (c)

  i.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 1.4)=1$,

  ii.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 12.3)=8/9$,

  iii.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 15)=7/9$,

  iv.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 40.5)=6/9$,

  v.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 55.7)=5/9$,

  vi.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 63.7)=4/9$,

  vii.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 68)=3/9$,

  viii.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 97.2)=2/9$,

  We can calculate other probabilities, for example, $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 60)=Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 63.7)+Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 68)=3/9+Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 97.2)=2/9=1/9+1/9+2/9=4/9$.
• For $\delta \ge 2.5$, $C_{\delta ,p}=\{v_2,v_3,v_5,v_6,v_7,v_8\}$. Then

  (a)

  $L{E}_{\delta }\left(p\right)=62.27$,

  (b)

  i.

  $Prob(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=15)=1/6$,

  ii.

  $Prob(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=68)=1/6$,

  iii.

  $Prob(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=40.5)=1/6$,

  iv.

  $Prob(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=97.2)=2/6$,

  v.

  $Prob(p\mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}=55.7)=1/6$,

  (c)

  i.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 15)=1$,

  ii.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 40)=5/6$,

  iii.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 55.7)=4/6$,

  iv.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 68)=3/6$,

  v.

  $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 97.2)=2/6$.

  Similarly, we can calculate the probabilities $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 45)=4/6$, $Pro{b}_{\delta }(L{E}_{\delta }\left(p\right)\ge 100)=0$.

In contrast with the statistical solutions used in [14,15,16], we adopted a probabilistic framework that gives more precise evaluations of probabilities. The SAIN algorithms also include some statistically established methods, such as the t-test, for ranking variables before applying the inference method.

4. SAIN: A Modular Diagram and Functional Information Flow

The SAIN framework consists of the following modules (Figure 1):

• Multimodal data of a new object X.
• An existing repository D of multimodal data of many objects, labelled with their outcome.
• A module of algorithms for searching in the database D and based on the distance between X and each object in D.
• Defining a subset $D_x$ from D, so that X is closer to the objects in $D_x$ based on a given threshold.
• A module of algorithms for building a model $M_x$ in $D_x$.
• An inference algorithm to derive the output for X from the model $M_x$ and to visualise it for explanation purposes. Figure 1 gives a modular view of the SAIN framework and Figure 2 shows the information processing flow:

  (a)

  Encoding the multimodal data of X and D. -

  (b)

  Choosing a distance matrix and similarity search in the data set D.

  (c)

  Calculating the aggregated difference between the new data vector X and the closest vectors in $D_x$.

  (d)

  Creating a model $M_x$ in $D_x$.

  (e)

  Applying inference by calculating the $X_{c,j}$ for each class $C_j$ (or output value), using the wwkNN method in [5].

  (f)

  Reporting and visualisation of results of the individual model Mx. This is illustrated in Figure 3.

The inference method is based on the wwkNN (weighted variables, weighted samples k-nearest neighbour) proposed by Kasabov [7]. This method first ranks the impact of the variables (multimodal ones) to estimate their weights/impact towards the output using a t-test; then, it measures the distance between the new object X and the ones in the database $D_x$ and weighs it. For each class $C_j$, the higher a variable is ranked, the closer the samples/objects belong to class $C_j$. to X, the higher the calculated value of $C_j$ is. The new object X is classified in class $C_l$ if $X_{cl}$ is the highest among all $X_{cj}$ values In Figure 1 and Figure 2, we present the modular diagram and the functional information flow of SAIN.

Figure 1. A modular diagram of the proposed SAIN computational framework

Figure 1. A modular diagram of the proposed SAIN computational framework

Figure 2. A flow of data and information processing in the SAIN computational framework

Figure 2. A flow of data and information processing in the SAIN computational framework

Figure 3. An example of visualisation of a personalised SAIN model. The closest four samples (out of 6) to the new object (star) are from class 1 (in red) using the top three informative variables. Each sample is a multimodal one, and the top 3 variables can be of different modalities

Figure 3. An example of visualisation of a personalised SAIN model. The closest four samples (out of 6) to the new object (star) are from class 1 (in red) using the top three informative variables. Each sample is a multimodal one, and the top 3 variables can be of different modalities

5. Case Studies for Medical Diagnosis and Prognosis

We present three case studies in which we applied SAIN.

5.1. Heart Disease Diagnosis

We worked with the well-known Cleveland dataset, which contains multiple data types [21]. The UCI Heart Disease data set includes 76 attributes. As in most articles, the attributes in our experiment data were restricted to 14, see Table 22.

The problem is a binary classification of whether the patient has or does not have heart disease.

First, we selected suitable distance metrics and weights to classify the attributes. For binary objects, the distance metric is simply whether they are equal; for non-binary discrete objects such as resting electrocardiographic results, the appropriate distance measure is not obvious and should be informed by an expert. We give electrocardiographic results 0 for normal, 1 for having ST-T wave abnormality, and 2 for showing probable or definite left ventricular hypertrophy following Estes’ criteria.

Many studies with the Cleveland dataset have been tested with different machine learning techniques. For example, [21] lists different algorithms and performances ranging from 47% to 80% accuracy. SAIN achieved an 82% accuracy score. Why SAIN? The search is fast, uses appropriate distances chosen by a medical expert, and provides explainability at a personal level, including probabilities. It offers different scenarios for modelling by experimenting with different sets of features, parameters, and preferred outcome visualizations.

The SAIN experiment used binary and numerical representation for each variable as described. We have used the same data representation (recommended by medical experts) as in the original paper [21]. The accuracy of the SAIN experiment was 82%, the same accuracy as in [6], which used classical machine learning. In addition, SAIN allows visualising each personalised model as shown in the examples in Figure 3 and Figure 8.

5.2. Time Series Classification

The proposed SAIN framework can incorporate time series data, as another modality, in addition to other modalities of data for a person, making a joint multimodal personal vector. A time series data is encoded into binary vector by using spike encoding algorithms [6], where if there is a positive change from one discrete time point to the next one in the time series, there will be a positive spike( encoded as 1); a negative change will result in a negative spike (-1) and no change will result in 0 value. This is illustrated on a hypothetical time series in Figure 4. This approach applies to any time series raw data, at any time scales, and here we show just two hypothetical examples of brain EEG data (Figure 5) and cardio data (Figure 6).

Many data sets for classifying outcomes of events consist of multiple time series. Each variable in a time series may depend on other variables that change in time. The proposed model can deal with this problem by encoding time series (signal) into binary vectors, which can be processed for classification in the SAIN framework. The variables for this data set are 14 channels of temporal EEG data, located at places of interest on the human scalp.

The signals measured over the same period are the EEG channels, fMRI voxels, ECG electrodes, seismic sensory signals, financial time series, gene expressions, voice, and music frequency bands [11]. Even when the variable (signal) measurements are independent, the signals may impact each other as they represent the same object/person at the same time period. The number N of these signals can vary from just a few for a short time window T (Figure 4) to hundreds and thousands when the time varies from a few milliseconds to minutes, hours, days, etc.

Figure 5 shows an EEG experiment, and Figure 6 shows a cardio-vascular disease signal.

Next, we present a simple example of how this search can be computed for a new record X consisting of only three variables/signals (e.g., EEG channels, ECG electrodes) over a short period of 5 time moments and the database D consisting of only six such records, which are labelled by outcome labels 1,2,3 (e.g., diagnosis, prognosis).

In addition to the record X, a weight vector is supplied with the weighted importance of the signals at different time points, e.g. $W=[0.1,0.2,0.4,0.2,0.1]$, meaning that the most important and informative part of the measurements is at time point 3.

The new record $X=[1,1,-1,0,1]$ (signal, EEG channel 1) 0, 1, 1, 1, -1 (signal, EEG channel 2) 1, 1, -1, -1, 0 (signal, EEG channel 3), $W=[0.1,0.2,0.4,0.2,0.1]$

The database contains records $(Records,R1,R2,R3,R4,R5,LabelsL)$ where (see Table 23):

The new record X of EEG-signals will be classified in class 1 as it is closest according to the Euclidean distance, class 1 data samples $R1$ and $R2$.

5.3. Predicting Longevity in Cardiac Patients

We utilised a data set [22] in which we applied a binary classification on whether the patient had an event (e.g., death) and further to those that had an event, whether this would occur in the near future (within the next 180 days, e.g., approximately six months). The data set contained a set of 150 variables and an outcome, with 295 patients in the first data set and 49 in the second. The data included a mix of variables that could be grouped as follows:

• demographics, risk factors, disease states, medication and deprivation scores,
• echocardiography, cardiac ultrasound measurements,
• advanced ECG measurements,

The other data includes the days until the event occurred and the censor date for the Cox proportional hazard monitoring.

The objectives are to predict an arrhythmic event or death.

Before running the algorithm, the data was normalised, and to account for the data being unbalanced, we utilised the SMOTE data balancing method [11] each time we left one out (ensuring that we did not SMOTE when the true data point was part of the data set). For the event classification data set, the model achieved an accuracy of 79%. This is broken down into classifying no event (198/247, 80%) and an event with (36/49, 73%) accuracy. It is worth noting that the confidence of each individual could be explored with a sample of the confidence for classification in Figure 7.

For the second experiment, we normalised the dataset and removed any columns with unknown values. We then applied a genetic algorithm to find the set of features to use for classification. We found a set of 34 variables which would provide an accuracy of $81\%$ with (34/34) for class 0 and (6/15) for class 1. Alternatively, if we apply SMOTE and focus more on the accuracy of class 1, we obtain $69\%$ accuracy, however, more evenly distributed with (24/34) for class zero and (10/15) for class 1.

Experiment one, in which we used a non-balanced complete data set, showed satisfactory results of 80% for class 0 (no event) and 73% for class 1 (event). This demonstrates the ability of the SAIN method to work with imbalanced data. It also shows the superiority of utilising the missing values rather than removing them in experiment two. Furthermore, the selection of variables with a genetic algorithm also showed improvement. The genetic algorithm, included in the SAIN software, can also help select biomarker variables in other cases (See Figure 8).

Figure 8. An individual model of survival showing the closest neighbouring samples in the top 3 ranked variables

Figure 8. An individual model of survival showing the closest neighbouring samples in the top 3 ranked variables

6. Data and Software Availability

The data has been obtained from the UCI Cleveland from https://archive.ics.uci.edu/dataset/45/heart+disease; the EEG data is available from https://github.com/KEDRI-AUT/NeuCube-Py/tree/master/example_data. Access to the software is available on request.

7. Conclusions

The paper presents a new search and inference method, called SAIN, for multi-modal data integration and personalised model creation based on these multi-modal data. The model not only evaluates the outcome for a person more accurately than traditional machine learning methods using a single modality of data, but it also explains the proposed solution in terms of probability and visual explanation.

In its current form, the paper is more directed towards revealing a new methodology and algorithms than real full-scale medical applications. However, we have illustrated the methods using hypothetical and real case health and medical data sets. Further utilisation of the proposed framework is currently being developed for large-scale biomedical data.

The proposed new method offers new functionality and features for personalised search and model creation in multimodal data, some of which are listed below:

• The method is suitable for multimodal data searches in heterogeneous data sets, e.g., numbers, text, images, sound, categorical data;
• It is suitable for personalised model creation to classify or predict specific outcomes based on multimodal and heterogeneous data.
• It uses a similarity measure based on multicriteria metrics. In this way, inaccurate measurement of similarity on a large number of heterogeneous variables is avoided.
• Its search is fast even on large data sets and includes advanced personalised searches with multiple parameters and features;
• It facilitates multiple solutions with corresponding probabilities;
• It is suitable for unsupervised clustering in multimodal heterogeneous data.

The proposed method is implemented as a computer system and applied to several case studies to illustrate its advantages and applicability. The SAIN method described in Section 4 was implemented as a software system.

In conclusion, integrating all possible data modalities for a single subject to predict/classify the object’s state in relation to existing ones is an open problem in data science. While the creation of personalised models based on a single modality data [23] and clustering of single modality data into a single cluster [24] have been successfully developed, the theory, framework and algorithms, proposed in this paper, are the first to integrate all data modalities for a single subject together into a single vector-based representation and to make an inference based on it. For the first time, time series, such as EEG and ECG data, are included in this unified representation, after suitable encoding. In this respect, spike encoding of time series is used, integrating statistical and brain-inspired information representation. The human brain integrates sensory data modalities into its spatio-temporal structure, and brain-inspired models using spike information representation have already been developed for learning [6,11,25] and for explanation of the learned patterns [26]. However, brain-inspired computers are still in their early stage of development [11], and even if they are developed, they may not be able to integrate all possible modalities of data into one brain-inspired mode. This paper offers a solution to the problem of multimodal personalised data integration and inference, six novel features characterise that: (1) it includes all possible modalities of data; (2) it can be implemented on any conventional computer platforms; (3) it takes into account the differences across modalities of data through offering different distance measures; (4) it offers a new way of ranking existing multimodal objects in order of similarity to a new multimodal object and uses that for building multiple neighbourhood clusters; (5) it offers a probability based inference with the use of the different similarity clusters; (6) it explains the inferred results, both in terms of probabilities and visual representation. The proposed original method here is planned to be applied to large-scale multimodal data for biomedical and health applications in the future.

The proposed method is implemented as a computer system and applied to several case studies to illustrate its advantages and applicability. The SAIN method described in Section 4 was implemented as a software system.