Evaluation Metrics in Learning Systems: A Survey

1. Introduction

Machine Learning and Deep Learning, having become the twin pillars of artificial intelligence, have attracted attention in recent decades owing to their ability to solve complex problems and automate solutions in many fields, including Natural Language Processing [1], Computer Vision [2], Speech Recognition [3], Recommender Systems [4],medicine [5,6] to signal processing [7], video processing [8], natural language processing [9], large language modeling [10], virtual reality [11], time-series analysis [12,13,14], and educational systems [15]. However, with the development and tuning of a machine learning model, it extends beyond the design of algorithms and training of the algorithm itself; not only are these important steps, but the accurate and systematic evaluative assessment of the performance of these models is, in fact, one of the key important steps in the development process. Evaluation metrics assist model builders in measuring quantifiably and qualitatively the strengths and weaknesses of a model [16]. These metrics have become vital to secure the reliability, accuracy, and efficiency of a model’s differential applications in the real world [17]. A model may perform very well in a hypothetical world; however, assuming no accurate metric is at hand, one cannot confidently claim that it will also do well in the real world or when confronted with newer data. Evaluation metrics allow the researchers to investigate the model from various perspectives, such as its generalization property, resistance to overfitting, and behaviour on unbalanced or noisy data. These are important attributes for a developer to consider while making design decisions that affect the improvement, learning procedure, or even choice compared to a more exemplary architecture. Adjusting parameters, providing rewards in training data-choice options to weigh their values, can substantially affect the final performance.

In addition, evaluation metrics serve as a common language among researchers [18]. In commercial projects, these metrics help businesses evaluate the value of a model from an economic and practical perspective. For example, for an image-based medical diagnosis system, the degree of correctness in disease diagnosis or False Positive Rate may directly influence the lives of patients. For text processing applications such as machine translation systems, a metric like BLEU or ROUGE may determine the quality of translation or summarization. Therefore, the choice of the right metric depends not only on the type of problem, but also on the final application of using the model. The evaluation measures used are very diverse depending on the type of problem, i.e., classification, regression, clustering, content creation, etc., and the domain of use, i.e., text, image, audio, etc. There are numerous issues in text processing, including text classification (opinion analysis, sentiment analysis, spam classification), text generation, text summarization, and machine translation. Accuracy, Precision, Recall, and F1-score are prevalent metrics for classification problems. BLEU for translation, ROUGE for summarization, or Perplexity for language models are standard metrics. These metrics help to evaluate the quality of the model’s responses in terms of semantics, structure, and compliance with human natural language. In image-related problems, such as object recognition, segmentation, and image classification, metrics such as Accuracy, Intersection over Union (IoU), Mean Average Precision (mAP), and Top-k Accuracy are very widely used. Likewise, for speech processing, there are metrics such as Word Error Rate (WER), which are used to evaluate speech-to-text systems. For recommender systems, one uses a metric such as Mean Reciprocal Rank (MRR) to measure the quality of the recommendation ranking. Each of these metrics provides various information about the model’s performance, depending on the type of problem and the application objective. One of the main problems with the use of evaluation metrics is the choice of a metric that most accurately reflects the project goal [19]. In an unbalanced data classification problem, accuracy is misleading on its own, since the model can predict the majority class and perform well, but perform badly in the detection of the minority class. In such cases, metrics such as the F1-score or Area Under the ROC Curve (AUC-ROC) provide more accurate information. Specific metrics are also not calibrated to the end-user’s goals. For example, for a machine translation system, a high BLEU score may indicate structure similarity to the reference text, but does not guarantee the semantic quality of the translation. For this reason, human evaluation is sometimes required alongside automated metrics.

This work aims to provide a comprehensive review of learning system evaluation metrics. Machine learning evaluation metrics are not only a mechanism to measure the performance of models, but also a recipe to improve, compare, and select the optimal models. By providing quantitative and qualitative data, these metrics help developers and researchers construct effective, reliable, and feasible models. Since machine learning problems and domains are different, the use of the right metric is dependent upon a deep knowledge of problem, data, and project goals.

2. Text Mining

Text mining is a nascent and growing field that tries to extract meaningful information from unstructured and structured natural texts. Text Mining converts unstructured text into a structured format to identify meaningful patterns and new insights [20]. By using advanced analytical techniques, such as machine learning algorithms and deep learning algorithms, it is possible to discover hidden relationships in unstructured data. Text is one of the most common data types in a database. These data can be divided into three categories [21]:

• Structured Data: These data are standardized as tables of multiple rows and columns. This makes them easier to store and process. Structured data can include entries such as name, address, and phone number.
• Unstructured Data: These data do not have a predefined and specific format. This data can include text from social media, product reviews, or video and audio files.
• Semi-structured Data: As the name suggests, this data combines structured and unstructured data. Examples of semi-structured data include XML, JSON, and HTML files.

Since nearly 80% of the organizational data in the world is unstructured, text mining is considered a very valuable practice in organizations [22]. Text mining tools and techniques allow us to convert unstructured documents into a structured format for high-quality analysis and insight. This, in turn, improves organizational decision-making and leads to better business outcomes [23]. Text mining includes various applications such as marketing applications, Enabling better CRM [24], security applications [25], deception detection [26], medicine and biology [27], literature-based gene identification [28], academic applications, and research stream analysis. It also includes text mining in the natural language processing of users, such as Named-entity recognition [29], Question answering [30], Automatic summarization [31], Natural language generation and understanding [32], Machine translation [33], Foreign language reading and writing [34], Speech recognition, and text proofing [35], and optical character recognition [36]. Different evaluation criteria depending on the user in text mining are defined in the literature. In this section, an attempt has been made to introduce the most critical evaluation criteria in each field.

2.1. Text Data Augmentation

Recent research on data augmentation for natural language processing, including GAN neural networks, has provided various technological advances. Frameworks, tools, and implementation of text-enhanced data pipelines are promoted. However, the lack of evaluation criteria and methods comparison standards, for various reasons including data and criteria, has made the comparison meaningless. Also, there is a lack of standardization of methods, and textual data enhancement research uses unified criteria to compare different enhancement methods. Therefore, researchers seek relevant evaluation criteria for textual data augmentation techniques. Continuing this section, we have tried to categorize the most important criteria for increasing data for natural language processing [37].

2.1.1. Machine Translation Quality

Human evaluation is the best option for evaluating translation quality in intelligent systems and is considered the ground truth. However, since human evaluation is time-consuming and expensive, it is only possible to evaluate some new output of the model by a group of people. For this purpose, a group of evaluation metrics have been introduced that make it possible to evaluate the models to some extent. In the following, we discuss some of the most important metrics [37].

BLEU Score

The BLEU (Bilingual Evaluation Understudy) score is an automated metric for evaluating the quality of text generated by machine translation (MT) systems by comparing it to one or more reference translations [38]. BLEU assesses how closely a machine-generated translation matches professional human translations, focusing on both accuracy and fluency.

 Definition 1. 

Mathematically, the BLEU score is defined as:

$$\mathit{BLEU}=\mathit{BP}\times exp\left(\sum _{n=1}^N{w}_{n}log{p}_n\right)$$

(1)

where:

• N is the maximum n-gram length considered (commonly $N=4$).
• $w_n$ is the weight assigned to the precision score of n-grams of order n (typically uniform weights, $w_n={\displaystyle \frac{1}{N}}$).
• $p_n$ is the modified precision for n-grams of order n.
• BP is the brevity penalty that penalizes overly short translations.

Modified n-gram Precision ($p_n$) The modified n-gram precision $p_n$ evaluates translation quality across a multi-sentence test corpus. For each source sentence, which may have multiple reference translations, we compute n-gram matches against all its references. The final precision is calculated by aggregating counts across all sentences in the corpus.

$$p_n={\displaystyle \frac{{\sum }_{C\in \left\{\mathrm{Candidates}\right\}}{\sum }_{\mathrm{n}\text{-}\mathrm{gram}\in C}{\mathrm{Count}}_{\mathrm{clip}}\left(\mathrm{n}\text{-}\mathrm{gram}\right)}{{\sum }_{C^{\prime }\in \left\{\mathrm{Candidates}\right\}}{\sum }_{\mathrm{n}\text{-}\mathrm{gram}\text{’}\in C^{\prime }}\mathrm{Count}\left(\mathrm{n}\text{-}\mathrm{gram}\text{’}\right)}}$$

(2)

where:

• $\left\{\mathrm{Candidates}\right\}$ represents the set of all candidate translations in the test corpus
• ${\mathrm{Count}}_{\mathrm{clip}}\left(\mathrm{n}\text{-}\mathrm{gram}\right)$ is computed as:

  $${\mathrm{Count}}_{\mathrm{clip}}\left(\mathrm{n}\text{-}\mathrm{gram}\right)=min({\mathrm{Count}}_C\left(\mathrm{n}\text{-}\mathrm{gram}\right),\underset{R\in \mathrm{Refs}\left(C\right)}{max}{\mathrm{Count}}_R\left(\mathrm{n}\text{-}\mathrm{gram}\right))$$

  (3)

  where $\mathrm{Refs}\left(C\right)$ are the reference translations for the candidate C
• $\mathrm{Count}\left(\mathrm{n}\text{-}\mathrm{gram}\text{’}\right)$ counts each n-gram occurrence in the candidate translation

Brevity Penalty (BP) The brevity penalty penalizes candidate translations that are shorter than the reference translations to prevent overly concise translations that achieve high precision by omitting necessary content. It is calculated as follows:

$$\mathrm{BP}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}c>r\hfill \\ e^{(1-{\displaystyle \frac{r}{c}})}\hfill & \mathrm{if}c\le r\hfill \end{array}\right.$$

where:

• c is the total length (in words) of the candidate translations.
• r is the effective reference length, defined as the sum of the lengths of the reference translations closest to each candidate sentence’s length.

Logarithmic Formulation The BLEU score can also be expressed in the logarithmic domain for clarity:

$$log\left(\mathrm{BLEU}\right)=min\left(1-{\displaystyle \frac{r}{c}},0\right)+\sum _{n=1}^N{w}_{n}log{p}_n.$$

(4)

Note: BLEU scores range from 0 to 1, where a higher score indicates greater similarity to the reference translations. In practice, BLEU scores are often multiplied by 100 to represent them as percentages.

2.1.2. Text Generation Quality

Text generation is the task of producing text similar to human-generated texts with the goal of being indistinguishable by humans. This task is formally known in the literature as “natural language generation”.

Text generation can be investigated with Markov processes or deep generative models such as GANs. The continuation of the most important metrics for evaluating the quality of text generation is given.

• Novelty:Novelty and Diversity are distinct. Novelty covers the difference of the produced text from the training set, while Diversity covers the difference of the produced text from other produced texts. A model can be diverse but not novel. These two criteria are listed below [39]:

  $$Novelty\left(S_i\right)=1-max{\left\{\phi (S_i,C_j)\right\}}_{j=1}^{j=|C|}$$

  (5)
• Diversity: BLEU [38] and Self-BLEU [40] are common metrics for quality and diversity evaluation, respectively:

  $$E_{C\sim Q}\mathrm{NSBLEU}\left(C\right)=-\sum _{i=1}^N{Q}_i\left(1-{(1-Q_i)}^{|C|-1}\right)$$

  (6)
  Calculate BLEU score for every generated sentence, and define the average BLEU score to be the Self-BLEU of the document.
  Discriminative BLEU [41]: For tasks with inherently diverse outputs, such as conversational response generation, where there is a wide range of plausible and acceptable responses, $\Delta$BLEU is introduced as an enhanced version of the traditional BLEU score. Unlike standard BLEU, $\Delta$BLEU incorporates human judgments about the quality of reference texts, making it more discriminative. It rewards matches with high-quality references and penalizes matches with low-quality ones, thereby better aligning with human evaluations. This makes $\Delta$BLEU particularly suitable for tasks like conversational response generation, where the space of valid outputs is vast and subjective. The $\Delta$BLEU score for a corpus is computed as:

  $$\Delta \mathrm{BLEU}=\mathrm{BP}·exp\left(\sum _{n=1}^{N}log{p}_n\right)$$

  where BP is the brevity penalty, and $p_n$ is the modified n-gram precision for n-grams of order n. The modified n-gram precision $p_n$ is defined as:

  $$p_n={\displaystyle \frac{{\sum }_i{\sum }_{g\in \mathrm{n}\text{-}\mathrm{grams}\left(h_i\right)}{max}_{j:g\in r_{i,j}}{w}_{i,j}·{\#}_g(h_i,r_{i,j})}{{\sum }_i{\sum }_{g\in \mathrm{n}\text{-}\mathrm{grams}\left(h_i\right)}{max}_j{w}_{i,j}·{\#}_g\left(h_i\right)}}$$

  where:

  −

  i is index over the test set, and j is index over the set of references $r_{i,j}$ for the i-th input message.

  −

  ${\#}_g(h_i,r_{i,j})$ is the minimum number of occurrences of n-gram g in the hypothesis $h_i$ and reference $r_{i,j}$.

  −

  $w_{i,j}$ is the human-assigned weight for reference $r_{i,j}$.

• Fluency: This is a subjective and multidimensional concept that can be related to various criteria, such as readability, clarity, and naturalness of the produced text. The context, purpose and audience of the text production task influence fluency. For example, a fluent text for a news article may differ from a fluent text for a children’s story. The text production fluency depends on the desired program. In technical writing, clarity and precision can be essential. The following evaluation criteria can be used to evaluate fluency [42].
  SLOR:
  The Syntactic Log Odds Ratio (SLOR) is a metric designed for the evaluation of fluency in natural language generation (NLG) outputs. Its primary purpose is to assess the fluency of generated sentences without relying on reference sentences, which can be particularly useful in scenarios where references are unavailable or impractical to obtain. The SLOR metric is defined mathematically as follows [43]:

  $$\mathrm{SLOR}\left(S\right)={\displaystyle \frac{1}{|S|}}\left(ln{p}_M\left(S\right)-ln{p}_u\left(S\right)\right)$$

  (7)
  In this equation, S represents the sentence being evaluated, $p_M\left(S\right)$ is the probability assigned to the sentence by a language model (LM), and $p_u\left(S\right)$ is the unigram probability of the sentence, calculated as the product of the probabilities of each individual token in the sentence. The normalization by $|S|$, the length of the sentence, ensures that the metric accounts for variations in sentence length, allowing for a fair comparison between sentences of different sizes. By leveraging the probabilities from a trained language model, SLOR provides a more nuanced evaluation of fluency, making it a valuable tool for researchers and practitioners in the field of natural language processing.
  Perplexity(PPL),wang2022perplexity: Perplexity (PPL) is one of the most common criteria for evaluating language models. PPL is a metric used to evaluate the fluency and quality of generated text by language models. It is derived from cross-entropy. Mathematically, the cross-entropy H for a sequence of tokens $t_1,t_2,\dots ,t_m$ can be expressed as:

  $$H=-\sum _{i=1}^{m}logP(t_i\mid t_1,t_2,\dots ,t_{i-1}),$$

  where $P(t_i\mid t_1,t_2,\dots ,t_{i-1})$ is the predicted probability of the i-th token given the previous tokens. PPL is then defined as the exponentiation of the average cross-entropy:

  $$PPL\left(s\right)=exp\left({\displaystyle \frac{H}{m}}\right)=exp\left(-{\displaystyle \frac{1}{m}}\sum _{i=1}^{m}logP(t_i\mid t_1,t_2,\dots ,t_{i-1})\right),$$

  (8)

  where s is the input sentence and m is the number of tokens in the sequence. A lower PPL value indicates better fluency and quality of the text generated by the model.
• Sentiment Consistency: In the context of text generation, maintaining sentiment consistency is crucial for ensuring that the generated text aligns with the expected emotional tone of the input prompts. In [45], two primary metrics are introduced to evaluate sentiment consistency in generated continuations: SENTSTD and SENTDIFF. These metrics help assess how well the sentiment of the generated text corresponds to the ground truth.
  • SENTSTD (Sentiment Standard Deviation),feng2020genaug: This metric measures the average standard deviation of sentiment scores for each batch of generated continuations, concatenated with their respective input prompts. By calculating the standard deviation across all test examples, a lower SENTSTD value indicates greater consistency in sentiment among the generated continuations. This suggests that the generated texts exhibit a more stable emotional tone.
  • SENTDIFF (Sentiment Difference),feng2020genaug: This metric evaluates the average difference in sentiment scores between each batch of continuations and their corresponding ground-truth reviews. By averaging these differences across all test examples, a lower SENTDIFF value indicates that the sentiment of the generated continuations aligns more closely with the ground-truth reviews. This metric is essential for assessing the fidelity of the generated text to the expected sentiment.
• Semantic content preservation,zhang2019bertscore: This evaluation metric is designed to assess the semantic equivalence between generated text and reference text. Unlike traditional metrics that rely on surface form similarity, this metric leverages contextual embeddings to evaluate how well the meaning of the content is preserved in the generated output. The BERTSCORE metric computes the similarity between tokens in the candidate sentence and those in the reference sentence using cosine similarity of their contextual embeddings. The formula for computing the recall, precision, and F1 scores is given as follows:

  $$\begin{array}{cc}\hfill R_{BERT}& ={\displaystyle \frac{1}{|x|}}\sum _{x_i\in x}\underset{x_j\in \widehat{x}}{max}\mathrm{cosine}(e\left(x_i\right),e\left({\widehat{x}}_j\right))\hfill \end{array}$$

  (9)

  $$\begin{array}{cc}\hfill P_{BERT}& ={\displaystyle \frac{1}{|\widehat{x}|}}\sum _{{\widehat{x}}_j\in \widehat{x}}\underset{x_i\in x}{max}\mathrm{cosine}(e\left(x_i\right),e\left({\widehat{x}}_j\right))\hfill \end{array}$$

  (10)

  $$\begin{array}{cc}\hfill F_{BERT}& =2·{\displaystyle \frac{P_{BERT}·R_{BERT}}{P_{BERT}+R_{BERT}}}\hfill \end{array}$$

  (11)

  where:

  −

  x is the reference sentence tokenized into k tokens $\{x_1,x_2,\dots ,x_k\}$.

  −

  $\widehat{x}$ is the candidate sentence tokenized into l tokens $\{{\widehat{x}}_1,{\widehat{x}}_2,\dots ,{\widehat{x}}_l\}$.

  −

  $e\left(x_i\right)$ and $e\left({\widehat{x}}_j\right)$ represent the contextual embeddings of tokens $x_i$ and ${\widehat{x}}_j$, respectively.

  −

  $\mathrm{cosine}(a,b)$ is the cosine similarity between vectors a and b.

  This approach allows for a more nuanced evaluation of text generation, focusing on the preservation of meaning rather than mere lexical overlap, thereby addressing common pitfalls in traditional n-gram based metrics. To enhance this evaluation, Inverse Document Frequency (IDF) weighting is incorporated, which emphasizes the importance of rare and informative tokens. The Recall metric with IDF weighting is defined as follows:

  $$\begin{array}{cc}\hfill R_{BERT}& ={\displaystyle \frac{{\sum }_{x_i\in x}\mathrm{idf}\left(x_i\right)·{max}_{x_j\in \widehat{x}}\cos \mathrm{ine}(e\left(x_i\right),e\left({\widehat{x}}_j\right))}{{\sum }_{x_i\in x}\mathrm{idf}\left(x_i\right)}}\hfill \end{array}$$

  (12)

  where:

  −

  $\mathrm{idf}\left(x_i\right)=log\left({\displaystyle \frac{M}{n\left(x_i\right)}}\right)$ is the IDF score of token $x_i$, where M is the total number of reference sentences, and $n\left(x_i\right)$ is the number of sentences containing $x_i$.

2.1.3. Character n-gram matches

• Character n-gram F-score (CHRF),popovic2017chrf++: This evaluation metric assesses the quality of machine translation outputs by leveraging character n-grams. CHRF correlates well with human assessments, particularly for morphologically rich languages. It calculates precision and recall based on character n-grams, making it language-independent and robust against tokenization issues. The general formula for the n-gram based F-score is given by:

  $$\begin{array}{cc}\hfill {\mathrm{F}}_{\mathrm{CHRF}}\left(\beta \right)& ={\displaystyle \frac{(1+{\beta }^2)·{\mathrm{P}}_{\mathrm{CHRF}}·{\mathrm{R}}_{\mathrm{CHRF}}}{{\beta }^2·{\mathrm{P}}_{\mathrm{CHRF}}+{\mathrm{R}}_{\mathrm{CHRF}}}}\hfill \end{array}$$

  (13)

  where:

  −

  ${\mathrm{P}}_{\mathrm{CHRF}}$ is the character n-gram precision, calculated as the average precision across all n-grams from $n=1$ to N.

  −

  ${\mathrm{R}}_{\mathrm{CHRF}}$ is the character n-gram recall, representing the average recall across all n-grams from $n=1$ to N.

  −

  $\beta$ is a parameter that assigns $\beta$ times more weight to recall than to precision, with a typical value of 2 being optimal.

  The CHRF score is computed by averaging the precision and recall across n-grams of lengths from 1 to N, where N is typically set to 6 for character n-grams.
• CHRF+: CHRF+ enhances the original CHRF metric by incorporating word unigrams alongside character n-grams. This integration aims to improve correlation with direct human assessments by capturing both surface-level and semantic information. The precision and recall for CHRF+ (${\mathrm{P}}_{\mathrm{CHRF}+}$ and ${\mathrm{R}}_{\mathrm{CHRF}+}$) are calculated as the combined metrics of both character and word n-grams, averaged across all n-gram lengths.
• CHRF++: CHRF++ further refines the CHRF+ metric by optimizing the combination of character and word n-grams, particularly by incorporating word unigrams and bigrams. This variant enhances correlation with direct human assessments by capturing more contextual information from the text. The precision and recall for CHRF++ (${\mathrm{P}}_{\mathrm{CHRF}++}$ and ${\mathrm{R}}_{\mathrm{CHRF}++}$) are computed with enhanced weighting strategies for character and word n-grams, allowing for a more nuanced evaluation.

2.1.4. Prediction quality for classification

In the realm of machine learning and statistical analysis, the evaluation of binary classification models is crucial for understanding their performance and reliability. Various metrics have been developed to summarize the outcomes of these models, each offering unique insights into their predictive capabilities. This section focuses on several key metrics that are commonly employed to assess the quality of predictions in binary classification tasks.

The evaluation of a model’s performance typically begins with the construction of a confusion matrix, which categorizes the predictions into four distinct groups: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). These categories form the basis for calculating various performance metrics, which help researchers and practitioners gauge how well their models are performing.

Among the most widely used metrics are Accuracy, Precision, and Recall, each of which provides different perspectives on the model’s effectiveness. Accuracy offers a general measure of correctness, while Precision and Recall focus on the model’s performance concerning the positive class. Additionally, the Macro-F1 Score serves as a harmonic mean of Precision and Recall, providing a balanced evaluation, especially in scenarios with imbalanced class distributions.

Other important metrics include Jaccard Similarity, which quantifies the similarity between predicted and actual sets, and the Matthews Correlation Coefficient (MCC), which is particularly valuable for assessing performance on imbalanced datasets. Finally, the Area Under the Precision-Recall Curve (AUPRC), calculated using the trapezoidal rule, summarizes the model’s performance across various thresholds [48].

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}},$$

(14)

where: $TP$ = True Positives, $TN$ = True Negatives, $FP$ = False Positives, and $FN$ = False Negatives.

$$\mathrm{Precision}={\displaystyle \frac{TP}{\mathrm{Actual}\mathrm{Results}}}={\displaystyle \frac{TP}{TP+FP}}$$

(15)

Precision measures the accuracy of positive predictions.

$$\mathrm{Recall}={\displaystyle \frac{TP}{\mathrm{Predicted}\mathrm{Results}}}={\displaystyle \frac{TP}{TP+FN}}$$

(16)

Recall measures the ability of a model to find all the relevant cases (true positives).

$$\mathrm{Macro}\text{-}\mathrm{F}1\mathrm{Score}=2·{\displaystyle \frac{\mathrm{Total}\mathrm{Precision}·\mathrm{Total}\mathrm{Recall}}{\mathrm{Total}\mathrm{Precision}+\mathrm{Total}\mathrm{Recall}}}={\displaystyle \frac{{\sum }_{i=1}^{n}F{1}_{{\mathrm{score}}_i}}{n}}$$

(17)

Explanation: The Macro-F1 Score is calculated as the harmonic mean of total precision and total recall. It averages the F1 scores for each class, treating all classes equally regardless of their support (number of true instances).

$$\mathrm{Jaccard}\mathrm{Similarity}(A,B)={\displaystyle \frac{|A\cap B|}{|A\cup B|}}={\displaystyle \frac{|A\cap B|}{|A|+|B|-|A\cap B|}}$$

(18)

Explanation: The Jaccard Similarity measures the similarity between two sets A and B. It is defined as the size of the intersection divided by the size of the union of the sets, providing a value between 0 (no overlap) and 1 (complete overlap).

$$\mathrm{MCC}={\displaystyle \frac{TP·TN-FP·FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}}$$

(19)

Explanation: The Matthews Correlation Coefficient (MCC) is a measure of the quality of binary classifications. It considers all four categories of the confusion matrix (True Positives, True Negatives, False Positives, False Negatives) and provides a balanced metric that can be used even when classes are of very different sizes.

$${\mathrm{AUPRC}}_{\mathrm{trapezoid}}={\displaystyle \frac{1}{2}}\sum _{i=1}^n(r_{i+1}-r_i)(p_{i+1}+p_i)$$

(20)

Explanation: The Area Under the Precision-Recall Curve (AUPRC) calculated using the trapezoidal rule. Here, r represents the recall and p represents the precision at different thresholds. This metric summarizes the model’s performance across various levels of classification thresholds.

Confusion matrix: Confusion matrix is an $N\ast N$ matrix used to evaluate the performance of a classification model, where N is the number of target classes. The matrix compares the actual target values with the values predicted by the machine learning model. The main components of this matrix are shown in Figure 1, which can be expanded for more classes [49].Confusion matrices are the main tool for evaluating errors in classification problems. These graphical models encode the full specification of misclassifications. Confusion matrices are used for the following:

• Error inspection of each class.
• Setting software parameters such as detection threshold.
• Comparing software versions [50].

2.2. Dataset Correlations

How different datasets relate to each other can be examined by employing correlation and similarity measures. Specifically, we explore Spearman’s rank correlation coefficient to capture monotonic relationships in ordinal data and Cosine similarity to measure the orientation of vectors in high-dimensional space. These metrics provide insights into the strength, direction, and nature of the relationships among the datasets.

Spearman’s rank correlation coefficient: Spearman’s correlation coefficient ($r_s$) is a non-parametric measure used to assess the strength and direction of the association between two ranked variables. It is particularly valuable when dealing with ordinal data. Unlike Pearson’s correlation, which evaluates linear relationships, Spearman’s correlation focuses on monotonic relationships, making it suitable for a wider range of data types. The coefficient ranges from -1 to +1, where -1 indicates a perfect negative correlation, +1 indicates a perfect positive correlation, and 0 indicates no correlation. Spearman’s rank correlation coefficient is obtained by the following formula [51]:

$$r_s=1-6\sum {\displaystyle \frac{d^2}{n(n^2-1)}},$$

(21)

where

• $d_i$: The difference between the ranks of the two variables for each observation.
• n: The number of paired observations.

Cosine similarity: This metric is a widely used in machine learning and pattern recognition, particularly for measuring the similarity between high-dimensional vectors. It quantifies the cosine of the angle between two vectors, providing a measure of their orientation rather than their magnitude. This characteristic makes cosine similarity particularly useful in applications such as text analysis and image recognition, where the direction of the data points is often more significant than their magnitude.

The cosine similarity between two vectors $\overrightarrow{a}$ and $\overrightarrow{b}$ is defined mathematically as:

$$cos\theta ={\displaystyle \frac{\overrightarrow{a}·\overrightarrow{b}}{\parallel \overrightarrow{a}\parallel \parallel \overrightarrow{b}\parallel }}$$

(22)

where:

• $\overrightarrow{a}·\overrightarrow{b}$ is the dot product of the vectors $\overrightarrow{a}$ and $\overrightarrow{b}$.
• $\parallel \overrightarrow{a}\parallel$ is the Euclidean norm (magnitude) of vector $\overrightarrow{a}$, calculated as $\sqrt{{\sum }_{i=1}^d{a}_i^2}$.
• $\parallel \overrightarrow{b}\parallel$ is the Euclidean norm (magnitude) of vector $\overrightarrow{b}$, calculated as $\sqrt{{\sum }_{i=1}^d{b}_i^2}$.
• $\theta$ is the angle between the two vectors.

The resulting value of cosine similarity ranges from -1 to 1, where 1 indicates that the vectors are identical in orientation, 0 indicates orthogonality (no similarity), and -1 indicates that the vectors are diametrically opposed. [52]:

2.3. Automatic Speech Recognition (ASR) Performance

Automatic Speech Recognition (ASR) is a technology that enables computers to recognize and process human speech.

Word Error Rate (WER): WER quantifies the accuracy of the transcribed output by comparing it to a reference transcription. The word sequence hypothesized by the ASR system is aligned with the reference transcription, and the number of errors is computed as the sum of three types of errors: Substitutions (S): Words that are incorrectly recognized, Insertions (I): Extra words added to the ASR output that do not appear in the reference, and deletions (D): Words present in the reference transcription but missing from the ASR output [53]:

$$\mathrm{WER}={\displaystyle \frac{I+D+S}{N}}\times 100$$

(23)

Word Recognition Rate (WRR): Word Recognition Rate (WRR) is a metric used to evaluate the performance of Automatic Speech Recognition (ASR) systems. It measures the proportion of words correctly recognized by the ASR system compared to the total number of words in the reference text (ground truth). [54]:

$$WRR=1-WER={\displaystyle \frac{N-S-D-I}{N}}={\displaystyle \frac{H-I}{N}},$$

(24)

where $H=N-(S+D)$ represents Number of correctly recognized words.

2.4. Training

In this category, various metrics that evaluate the performance of machine learning algorithms are presented. The most important learning evaluation metrics include:

Error Rate: The error rate is defined as the complement of accuracy, providing a measure of the proportion of incorrect predictions made by the model. It is expressed mathematically as:

$$\mathrm{Error}=1-\mathrm{Accuracy}$$

(25)

Standard Binary Cross-Entropy Loss Function: The binary cross-entropy (BCE) loss function is utilized for binary classification tasks. It quantifies the difference between the predicted probabilities and the actual binary labels. The formula is given by [55]:

$$J_{bce}=-{\displaystyle \frac{1}{M}}\sum _{m=1}^M\left[y_{m}log\left(h_{\theta }\left(x_m\right)\right)+(1-y_m)log(1-h_{\theta }\left(x_m\right))\right]$$

(26)

where M is the number of training samples, $y_m$ is the target label for training sample m (0 or 1), $x_m$ is the input for training sample m, $h_{\theta }\left(x_m\right)$ is the predicted probability that the label is 1 for input $x_m$.

Weighted Binary Cross-Entropy: The weighted binary cross-entropy loss function adjusts the standard BCE to account for class imbalance by applying a weight to the positive class. The formula is [55]:

$$J_{wbce}=-{\displaystyle \frac{1}{M}}\sum _{m=1}^M\left[w·y_{m}log\left(h_{\theta }\left(x_m\right)\right)+(1-y_m)log(1-h_{\theta }\left(x_m\right))\right]$$

(27)

where w is the weight assigned to the positive class.

Standard Categorical Cross-Entropy: For single-label multiclass classification, the standard categorical cross-entropy loss function is expressed as [55]:

$$J_{cce}=-{\displaystyle \frac{1}{M}}\sum _{k=1}^K\sum _{m=1}^M{y}_m^{k}log\left(h_{\theta }(x_m,k)\right)$$

(28)

where K is the number of classes, $y_m^k$ is the target label for training example m for class k, $h_{\theta }(x_m,k)$ is the predicted probability for class k.

Standard Weighted Categorical Cross-Entropy: The weighted categorical cross-entropy loss function extends the standard categorical cross-entropy by incorporating class weights. It is defined as [55]:

$$J_{wcce}=-{\displaystyle \frac{1}{M}}\sum _{k=1}^K\sum _{m=1}^M{w}_k·y_m^{k}log\left(h_{\theta }(x_m,k)\right)$$

(29)

where $w_k$ is the weight for class k.

Real-World Weight Cross-Entropy (RWWCE): The RWWCE loss function is designed to account for the real-world costs associated with misclassification. It incorporates weights for false negatives and false positives, expressed as [55]:

$$J_{brwwce}=-{\displaystyle \frac{1}{M}}\sum _{m=1}^M\left[w_{mcfn}·y_{m}log\left(h_{\theta }\left(x_m\right)\right)+w_{mcfp}·(1-y_m)log(1-h_{\theta }\left(x_m\right))\right]$$

(30)

where $w_{mcfn}$ is the marginal cost of a false negative over a true positive, $w_{mcfp}$ is the marginal cost of a false positive over a true negative.

These metrics provide a comprehensive framework for evaluating the performance of classifiers in machine learning, particularly in scenarios involving imbalanced classes and significant costs associated with mislabeling.

2.5. Discriminator Metrics

Discriminator metrics are primarily used to evaluate and select the optimal solution in classification tasks. They play a crucial role in identifying the best model that can accurately predict future evaluations of a classifier [56]. Traditional metrics, such as accuracy, are widely used due to their simplicity and ease of interpretation. However, as highlighted in [56], accuracy often fails to provide sufficient discrimination power, particularly in imbalanced datasets where it tends to favor the majority class. To address these limitations, hybrid metrics such as Optimized Precision (OP) and Optimized Accuracy with Recall and Precision (OARP) have been proposed. In addition to hybrid metrics, ranking-based metrics such as Area Under the Curve (AUC) offer a holistic view of classifier performance. Table 1 summarizes these metrics, providing their formulas and highlighting their evaluation focuses.

2.6. Agreement Metrics

Agreement metrics are essential tools for evaluating the performance of classification models, particularly in clinical diagnostics and machine learning tasks. These metrics assess the level of agreement between predicted and actual outcomes, accounting for chance agreement. A 2x2 contingency table is often used to define these metrics, where a, b, c, and d represent the counts of true positives, false positives, false negatives, and true negatives, respectively. The total number of observations is given by $n=a+b+c+d$.

Raw agreement (I) measures the proportion of correctly classified instances ($I={\displaystyle \frac{a+d}{n}}$), while “Agreement Due to Chance” ($\widehat{I}$) estimates the proportion of agreement expected by random chance. The value of $\widehat{I}$ is computed as:

$$\widehat{I}={\displaystyle \frac{(a+c)(a+b)+(b+d)(c+d)}{n^2}}.$$

This value is crucial because it provides a baseline for evaluating whether the observed agreement is better than random guessing. Cohen’s kappa ($\kappa$) incorporates $\widehat{I}$ and is defined as:

$$\kappa ={\displaystyle \frac{I-\widehat{I}}{1-\widehat{I}}}.$$

Kappa thus quantifies the agreement between predictions and actual outcomes beyond what is expected by chance. A kappa score between 0.81 and 0.99 indicates near-perfect agreement [57,58].

Sensitivity ($Se={\displaystyle \frac{a}{a+c}}$) and specificity ($Sp={\displaystyle \frac{d}{b+d}}$) further evaluate the model’s ability to correctly classify positive and negative cases, respectively. These metrics, along with kappa, provide a comprehensive framework for assessing model performance in both clinical and machine learning applications. Table 2 summarizes these metrics and their definitions as discussed in [57,58].

2.7. Metrics for Evaluating Recommendation Systems

Recommender systems are an essential part of the largest known websites today. Without them, it will be difficult to find the right products and content for users [59]. These systems are known as a subset of NLP and ML. Recommender systems support users and developers of various computer and software systems to overcome information overload, perform information discovery tasks and approximate calculations, etc. These models have attracted a wide range of application scenarios from business process modeling to source code manipulation [60]. Due to this wide variety in application areas, different approaches and criteria have been adopted to evaluate them. In this section, we discuss the most important metrics for evaluating these models.

2.7.1. Predicting User Ratings

Recommendation systems often aim to predict how users rate items of interest. To evaluate the accuracy of these predictions, metrics such as Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) are commonly used [60]. These metrics compare the predicted ratings (${\widehat{r}}_{ui}$) with the actual ratings ($r_{ui}$) provided by users. Lower values for these metrics indicate higher accuracy. Additionally, normalized versions of these metrics are used to account for differences in rating scales.RMSE and MAE differ in how they penalize errors. RMSE penalizes larger errors more heavily due to squaring the residuals, making it more sensitive to outliers. If large errors are undesirable, RMSE is preferred. Both metrics can be normalized to account for the rating range, providing scaled versions that facilitate comparison across systems with different rating scales. These metrics provide a quantitative evaluation of the system’s ability to predict user ratings accurately.

Table 3. Metrics for Predicting User Ratings [60]

Metric	Formula and Description
Root Mean Squared Error (RMSE)	$$RMSE=\sqrt{{\displaystyle \frac{{\sum }_{(u,i)\in T}{({\widehat{r}}_{ui}-r_{ui})}^2}{N}}}$$ Measures the square root of the average squared differences between predicted (${\widehat{r}}_{ui}$) and actual ($r_{ui}$) ratings. Penalizes larger errors more heavily.
Mean Absolute Error (MAE)	$$MAE={\displaystyle \frac{{\sum }_{(u,i)\in T}|{\widehat{r}}_{ui}-r_{ui}|}{N}}$$ Computes the average absolute deviation of predicted ratings from actual ratings. All residuals are equally weighted.
Normalized RMSE (NRMSE)	$$NRMSE={\displaystyle \frac{RMSE}{r_{\max }-r_{\min }}}$$ Scales RMSE based on the range of ratings ($r_{\max }-r_{\min }$), enabling comparisons across systems with different rating scales.
Normalized MAE (NMAE)	$$NMAE={\displaystyle \frac{MAE}{r_{\max }-r_{\min }}}$$ Scales MAE similarly to NRMSE, making it comparable across systems with varying rating ranges.

Table 3. Metrics for Predicting User Ratings [60]

Metric	Formula and Description
Root Mean Squared Error (RMSE)	$$RMSE=\sqrt{{\displaystyle \frac{{\sum }_{(u,i)\in T}{({\widehat{r}}_{ui}-r_{ui})}^2}{N}}}$$ Measures the square root of the average squared differences between predicted (${\widehat{r}}_{ui}$) and actual ($r_{ui}$) ratings. Penalizes larger errors more heavily.
Mean Absolute Error (MAE)	$$MAE={\displaystyle \frac{{\sum }_{(u,i)\in T}|{\widehat{r}}_{ui}-r_{ui}|}{N}}$$ Computes the average absolute deviation of predicted ratings from actual ratings. All residuals are equally weighted.
Normalized RMSE (NRMSE)	$$NRMSE={\displaystyle \frac{RMSE}{r_{\max }-r_{\min }}}$$ Scales RMSE based on the range of ratings ($r_{\max }-r_{\min }$), enabling comparisons across systems with different rating scales.
Normalized MAE (NMAE)	$$NMAE={\displaystyle \frac{MAE}{r_{\max }-r_{\min }}}$$ Scales MAE similarly to NRMSE, making it comparable across systems with varying rating ranges.

2.7.2. Ranking-Based Metrics

Ranking-based metrics are essential for evaluating the performance of recommender systems. These metrics provide insights into how well a system can suggest relevant items to users based on their preferences. Below is a detailed overview of common ranking-based metrics, including Precision@K, Recall@K, Mean Reciprocal Rank (MRR@K), Mean Average Precision (MAP@K), Normalized Discounted Cumulative Gain (NDCG@K), and Hit Rate@K. Traditional classification criteria, such as accuracy, precision, recall, and f1, in the context of recommender systems can be used to see how many recommended items are relevant. But they cannot get the order in which the items are recommended. Customers have short attention spans - and it’s important to check if the right things are being recommended [61]. The $MAP@K$ metric stands for Mean Average Precision at K and is related to two aspects: 1) Are the predicted items related? 2) Are the most relevant items at the top? will pay. Recommendation quality measures for implicit feedback recommender systems can be calculated as follows:

Precision@K measures the proportion of relevant items among the top-K recommendations. It is calculated as the number of relevant items retrieved in the top-K divided by K,

$$Precision@K={\displaystyle \frac{1}{K}}\sum _{i=1}^{K}1\left(relevant\left(i\right)\right),$$

where K is The cutoff rank (i.e., the number of top-ranked items considered) and $1\left(relevant\right(i\left)\right)$ is an indicator function that returns 1 if the item at rank i is relevant, and 0 otherwise [62].

Recall@K:It is a metric used to evaluate the effectiveness of a recommendation system by measuring the proportion of relevant items successfully retrieved within the top-K recommendations. It is calculated as the number of relevant items retrieved in the top-K divided by the total number of relevant items:

$$Recall@K={\displaystyle \frac{1}{|R|}}\sum _{i=1}^{K}1\left(relevant\left(i\right)\right),$$

where $|R|$ is the total number of relevant items for the given user or query [63]. While $Precision@K$ evaluates the proportion of relevant items among the top-K recommendations, $Recall@K$ measures the proportion of all relevant items that are successfully retrieved within the top-K recommendations.

The Mean Reciprocal Rank (MRR) is a widely used evaluation metric in information retrieval and recommendation systems. It measures the effectiveness of a system in ranking relevant items higher in a list of recommendations. The importance of MRR lies in its ability to capture the rank of the first relevant item in a recommendation list. A higher MRR score indicates that the system consistently places relevant items closer to the top, thereby improving user satisfaction and engagement. The MRR for a set of Q queries or users is defined as the average of the reciprocal ranks of the first relevant item for each query. Let ${\mathrm{rank}}_i$ denote the rank position of the first relevant item for the i-th query. The formulation of MRR is given as:

$$\mathrm{MRR}={\displaystyle \frac{1}{Q}}\sum _{i=1}^Q{\displaystyle \frac{1}{{\mathrm{rank}}_i}},$$

(31)

The reciprocal rank, ${\displaystyle \frac{1}{{\mathrm{rank}}_i}}$, assigns higher scores to relevant items that appear earlier in the ranking. MRR aggregates these scores across all queries, providing a single metric that reflects the system’s ranking quality [64].

The Average Reciprocal Hit Rank at K (ARHR@k) is a ranking-based evaluation metric used to measure the effectiveness of recommendation systems. It focuses on the position of the first relevant item in the top-K recommendations for each user. ARHR@k extends the Mean Reciprocal Rank (MRR) by restricting the evaluation to the top-K items, making it particularly useful in scenarios where users are only likely to consider a limited number of recommendations.

The ARHR@k is calculated as the average of the reciprocal ranks of the first relevant item for each user, considering only the top-K recommendations. The formula is [64]:

$$\mathrm{ARHR}\text{@}\mathrm{k}={\displaystyle \frac{1}{U}}\sum _{u\in U}\sum _{i=1}^K{\displaystyle \frac{\delta (i,u)}{i}},$$

(32)

where:

• U is the total number of users,
• K is the cutoff rank,
• $\delta (i,u)$ is an indicator function that equals 1 if the item at rank i is relevant to user u, and 0 otherwise,
• i is the rank position of the item.

Mean Average Precision (MAP) is a widely used ranking-based evaluation metric in recommendation systems. It measures the quality of ranked results by considering both the relevance of items and their positions in the ranking. The Mean Average Precision (MAP) is defined as the mean of the Average Precision (AP) scores across all queries. For a single query q, the Average Precision (AP) is calculated as:

$$\mathrm{AP}\left(q\right)={\displaystyle \frac{1}{|R(q)|}}\sum _{k=1}^n\mathrm{Precision}\text{@}\mathrm{k}·\mathrm{Rel}\left(k\right)$$

(33)

where $|R(q)|$ is the total number of relevant items for query q, and n is the total number of recommended items. $\mathrm{Rel}\left(k\right)$ is a binary indicator function that equals 1 if the item at rank k is relevant to the query q, and 0 otherwise. The MAP is then computed as

$$\mathrm{MAP}={\displaystyle \frac{1}{Q}}\sum _{q=1}^Q\mathrm{AP}\left(q\right)$$

(34)

where Q is the total number of queries or users in the evaluation [64]. When the evaluation is constrained to the top K items in the ranked list, the metric becomes Mean Average Precision at K, denoted as MAP@K [65]. In this case, the summation in the AP formula is limited to the top K ranks:

$$\mathrm{AP}\text{@}\mathrm{K}\left(q\right)={\displaystyle \frac{1}{|R(q)|}}\sum _{k=1}^K\mathrm{Precision}\text{@}\mathrm{k}·\mathrm{Rel}\left(k\right)$$

(35)

$$\mathrm{MAP}\text{@}\mathrm{K}={\displaystyle \frac{1}{Q}}\sum _{q=1}^Q\mathrm{AP}\text{@}\mathrm{K}\left(q\right)$$

(36)

The key difference between MAP and MAP@K is that MAP considers the entire ranked list, while MAP@K evaluates only the top K items, making it more suitable for scenarios where users interact with a limited number of recommendations.

Normalized Discounted Cumulative Gain (nDCG@K) at K is a widely used metric in recommendation systems to evaluate the quality of ranked lists. It measures how well the predicted ranking of items corresponds to the ideal ranking. The DCG for a ranked list of K items is calculated as:

$$\mathrm{DCG}\text{@}\mathrm{K}=\sum _{i=1}^K{\displaystyle \frac{2^{{\mathrm{rel}}_i}-1}{{log}_2(i+1)}},$$

where ${\mathrm{rel}}_i$ is the relevance score of the item at position i. The nDCG@K is then computed as:

$$\mathrm{nDCG}\text{@}\mathrm{K}={\displaystyle \frac{\mathrm{DCG}\text{@}\mathrm{K}}{\mathrm{iDCG}\text{@}\mathrm{K}}},$$

The nDCG@K is obtained by normalizing the DCG with respect to the iDCG, ensuring that the metric ranges between 0 and 1, where 1 indicates a perfect ranking. where iDCG@K is the DCG of the ideal ranking for the same set of items [64].

2.7.3. Coverage

Coverage is a critical dimension in evaluating recommendation systems, as it measures the extent to which the system can make recommendations across the available information space. It refers to the proportion of available items or users for which the system can generate recommendations. This dimension is particularly important in scenarios where new items or users are introduced, or when there is insufficient data for certain items or users. Coverage is typically categorized into two main types: catalogue coverage, which evaluates the proportion of items in the system’s catalogue that can be recommended, and prediction coverage, which assesses the proportion of users or user interactions for which predictions can be made [60].

Catalogue Coverage

Catalogue coverage measures the proportion of items in the available catalogue that the system can recommend. It is calculated as follows [60]:

$$Catalogu{e}_{Coverage}={\displaystyle \frac{|S_r|}{|S_a|}}$$

(37)

where:

• $S_r$: The set of items recommended to users.
• $S_a$: The set of all available items in the catalogue.

Weighted Catalogue Coverage

Weighted catalogue coverage adjusts the catalogue coverage metric by considering the usefulness of the recommended items. It is defined as [60]:

$$WeightedCatalogu{e}_{Coverage}={\displaystyle \frac{|S_r\cap S_s|}{|S_s|}}$$

(38)

where:

• $S_r$: The set of items recommended to users.
• $S_s$: The set of items considered useful to users.

Prediction Coverage

Prediction coverage measures the proportion of users or user interactions for which the system can generate predictions. It is calculated as [60]:

$$Predictio{n}_{Coverage}={\displaystyle \frac{|S_p|}{|S_u|}}$$

(39)

where:

• $S_p$: The set of users for whom predictions can be made.
• $S_u$: The set of all users in the system.

Weighted Prediction Coverage

Weighted prediction coverage adjusts the prediction coverage metric by considering the usefulness of the recommendations for users. It is defined as [60]:

$$WeightedPredictio{n}_{Coverage}={\displaystyle \frac{{\sum }_{i\in S_p}f\left(i\right)}{{\sum }_{j\in S_u}f\left(j\right)}}$$

(40)

where:

• $S_p$: The set of users for whom predictions can be made.
• $S_u$: The set of all users in the system.
• $f\left(x\right)$: A function representing the usefulness of the recommendations for a specific user.

2.7.4. Diversity

Diversity [60,66] in recommendation systems measures the degree of variability or dissimilarity among the items in a recommendation list. High diversity ensures that the recommendations are not redundant and provides users with a broader exploration of the item space. A lack of diversity may result in repetitive or overly similar items, reducing the overall utility of the recommendation system. Several metrics have been introduced to quantify diversity, as described below.

2.7.4.1. Diversity

Diversity is calculated as the average dissimilarity between all pairs of items in a recommendation list. It is defined as [66]:

$$Diversity(c_1,\dots ,c_n)={\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=i}^n(1-Similarity(c_i,c_j))}{\frac{n·(n-1)}{2}}}$$

(41)

where:

• $c_1,\dots ,c_n$: Items in the recommendation list.
• $Similarity(c_i,c_j)$: A similarity metric between items $c_i$ and $c_j$.
• n: The total number of items in the recommendation list.

Similarity

The similarity between an item c and a target query t can be calculated using a weighted sum metric [66]:

$$Similarity(t,c)={\displaystyle \frac{{\sum }_{i=1}^n{\omega }_i·sim(t_i,c_i)}{{\sum }_{i=1}^n{\omega }_i}}$$

(42)

where:

• t: The target query.
• c: The item being compared.
• ${\omega }_i$: The weight assigned to the i-th feature.
• $sim(t_i,c_i)$: A similarity heuristic between the i-th feature of t and c.

Quality

To balance the trade-off between diversity and similarity, a quality metric is introduced. It combines the similarity of an item c with the target query t and the diversity of c relative to the items already selected in the recommendation list R [66]:

$$Quality(t,c,R)=Similarity(t,c)·RelDiversity(c,R)$$

(43)

where:

• $Similarity(t,c)$: The similarity of item c to the target query t.
• $RelDiversity(c,R)$: The diversity of c relative to the items in R.

Relative Diversity

Relative diversity measures the diversity of a new item c relative to the items already selected in the recommendation list $R=\{r_1,\dots ,r_m\}$. It is defined as [66]:

$$RelDiversity(c,R)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}R=\varnothing \hfill \\ {\displaystyle \frac{{\sum }_{i=1}^m(1-Similarity(c,r_i))}{m}},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(44)

where:

• c: The new item being evaluated.
• R: The set of items already selected in the recommendation list.
• $Similarity(c,r_i)$: The similarity between the new item c and an item $r_i$ in R.
• m: The number of items in R.

3. Image Mining

3.1. Evaluation Metrics for 3D Image Segmentation

3.1.1. Spatial Overlap Based Me3trics

• Basic cardinalities:

  $$m_{ij}=\sum _{r=1}^{|X|}{f}_g^i\left(x_r\right)f_t^j\left(x_r\right)$$

  (45)
• Generalization to fuzzy segmentations:

  $$TP=\sum _{r=1}^{|X|}min\left(f_t^1\left(x_r\right)f_1^g\left(x_r\right)\right)$$

  (46)

  $$FP=\sum _{r=1}^{|X|}max(f_t^1\left(x_r\right)f_1^g\left(x_r\right),0)$$

  (47)

  $$TN=\sum _{r=1}^{|X|}min\left(f_t^2\left(x_r\right)f_2^g\left(x_r\right)\right)$$

  (48)

  $$FN=\sum _{r=1}^{|X|}max(f_t^2\left(x_r\right)f_2^g\left(x_r\right),0)$$

  (49)
• Calculation of overlap based metrics:

  $$Dice={\displaystyle \frac{2|S_g^1\bigcap S_t^1|}{|S_g^1|+|S_t^1|}}={\displaystyle \frac{2TP}{2TP+FP+FN}}={\displaystyle \frac{2JAC}{1+JAC}}$$

  (50)

  $$jac={\displaystyle \frac{2|S_g^1\bigcap S_t^1|}{|S_g^1\bigcup S_t^1|}}={\displaystyle \frac{2|S_g^1\bigcap S_t^1|}{2(|S_g^1|+|S_t^1|-|S_g^1\bigcap S_t^1|)}}={\displaystyle \frac{Dice}{2-Dice}}$$

  (51)

  $$Recall=Sensitivity=TPR={\displaystyle \frac{TP}{TP+FN}}$$

  (52)

  $$Specificity=TNR={\displaystyle \frac{TN}{TN+FP}}$$

  (53)

  $$Fallout=FPR={\displaystyle \frac{FP}{FP+TN}}=1-TNR$$

  (54)

  $$FNR={\displaystyle \frac{FN}{FN+TP}}=1-TPR$$

  (55)

  $$Precision=PPV={\displaystyle \frac{TP}{TP+FP}}$$

  (56)

  $$FM{S}_{\beta }={\displaystyle \frac{({\beta }^2+1).PPV.TPR}{{\beta }^2.PPV+TPR}}$$

  (57)

  $$FMS={\displaystyle \frac{2.PPV.TPR}{PPV+TPR}}$$

  (58)

  $$E(S_t,S_g,x)={\displaystyle \frac{|R(S_t,x)\setminus R(S_g,x)|}{|R(S_t,x)|}}$$

  (59)

  $$GCE(S_t,S_g)={\displaystyle \frac{1}{n}}min\{\sum _i^{n}E(S_t,S_g,x_i),\sum _i^{n}E(S_g,S_t,x_i)\}$$

  (60)

  $$GCE={\displaystyle \frac{1}{n}}min\{{\displaystyle \frac{FN(FN+2TP)}{TP+FN}}+{\displaystyle \frac{FP(FP+2TN)}{TN+FP}},{\displaystyle \frac{FP(FP+2TP)}{TP+FP}}+{\displaystyle \frac{FN(FN+2TN)}{TN+FN}}\}$$

  (61)
• Overlap measures for multiple labels:

  $$JA{C}_{ml}={\displaystyle \frac{{\sum }_{labels,l}\alpha l{\sum }_{voxels,i}MIN(A_{li},B_{li}}{{\sum }_{labels,l}\alpha l{\sum }_{voxels,i}MAX(A_{li},B_{li})}}$$

  (62)

3.1.2. Volume Based Metrics

$$VS=1-{\displaystyle \frac{||S_t^1|-|S_g^1||}{|S_t^1|+|S_g^1|}}=1-{\displaystyle \frac{|FN-FP|}{2TP+FP+FN}}$$

(63)

3.1.3. Pair Counting Based Metrics

• Basic cardinalities

  $$a={\displaystyle \frac{1}{2}}\sum _{i=1}^r\sum _{j=1}^s{m}_{ij}(m_{ij}-1)$$

  (64)

  $$b={\displaystyle \frac{1}{2}}(\sum _{j=1}^s{m}_j^2-\sum _{i=1}^r\sum _{j=1}^s{m}_{ij}^2)$$

  (65)

  $$c={\displaystyle \frac{1}{2}}(\sum _{j=1}^r{m}_i^2-\sum _{i=1}^r\sum _{j=1}^s{m}_{ij}^2)$$

  (66)

  $$d=n(n-1)/2-(a+b+c)$$

  (67)

  $$a={\displaystyle \frac{1}{2}}[TP(TP-1)+FP(FP-1)+TN(TN-1)+FN(FN-1)]$$

  (68)

  $$b={\displaystyle \frac{1}{2}}[{(TP+FN)}^2+{(TN+FP)}^2-(T{P}^2+T{N}^2+F{P}^2+F{N}^2)]$$

  (69)

  $$c={\displaystyle \frac{1}{2}}[{(TP+FP)}^2+{(TN+FN)}^2-(T{P}^2+T{N}^2+F{P}^2+F{N}^2)]$$

  (70)

  $$d=n(n-1)/2-(a+b+c)$$

  (71)
• Generalization to fuzzy segmentations:
• Calculation of pair-counting based metrics:

  $$RI(S_g,S_t)={\displaystyle \frac{a+b}{a+b+c+d}}$$

  (72)

  $$ARI={\displaystyle \frac{{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{m_{ij}}{2}\right)-{\sum }_i\left(\genfrac{}{}{0pt}{}{m_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{m_j}{2}\right)/\left(\genfrac{}{}{0pt}{}{n}{2}\right)}{\frac{1}{2}[{\sum }_i\left(\genfrac{}{}{0pt}{}{m_i}{2}\right)+{\sum }_j\left(\genfrac{}{}{0pt}{}{m_j}{2}\right)]-{\sum }_i\left(\genfrac{}{}{0pt}{}{m_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{m_j}{2}\right)/\left(\genfrac{}{}{0pt}{}{n}{2}\right)}}$$

  (73)

  $$ARI={\displaystyle \frac{2(ad-bc)}{c^2+b^2+2ad+(a+d)(c+b)}}$$

  (74)

3.1.4. Information Theoretic Based Metrics

$$H\left(S\right)=-\sum _{i}p\left(S^i\right)logp\left(S^i\right)$$

(75)

$$H(S_1,S_2)=-\sum _{ij}p(S_1^i,S_2^j)logp(S_1^i,S_2^j)$$

(76)

$$p\left(S_g^1\right)=(TP+FN)/n$$

(77)

$$p\left(S_g^2\right)=(TN+FN)/n$$

(78)

$$p\left(S_t^1\right)=(TP+FN)/n$$

(79)

$$p\left(S_t^2\right)=(TN+FP)/n$$

(80)

$$p(S_i^i,S_2^j)={\displaystyle \frac{|S_1^i\bigcap S_2^j|}{n}}$$

(81)

$$p(S_1^1,S_2^1)={\displaystyle \frac{TP}{n}}$$

(82)

$$p(S_1^1,S_2^2)={\displaystyle \frac{FN}{n}}$$

(83)

$$p(S_2^1,S_1^2)={\displaystyle \frac{FP}{n}}$$

(84)

$$p(S_2^1,S_2^2)={\displaystyle \frac{TN}{n}}$$

(85)

$$MI(S_g,S_t)=H\left(S_g\right)+H\left(S_t\right)-H(S_g,S_t)$$

(86)

$$VOI=(S_g,S_t)=H\left(S_g\right)+H\left(S_t\right)-2MI(S_g,S_t)$$

(87)

3.1.5. Probabilistic Metrics

• Interclass Correlation (ICC):

  $$ICC={\displaystyle \frac{{\sigma }_s^2}{{\sigma }_s^2+{\sigma }_{ϵ}^2}}$$

  (88)

  $$ICC={\displaystyle \frac{M{S}_b-M{S}_w}{M{S}_b+(k-1)M{S}_w}}$$

  (89)

  $$M{S}_b={\displaystyle \frac{2}{n-1}}\sum _x{(m\left(x\right)-\mu )}^2$$

  (90)

  $$M{S}_w={\displaystyle \frac{1}{n}}\sum _x{(f\left(x\right)-m\left(x\right))}^2+{(f_t\left(x\right)-m\left(x\right))}^2$$

  (91)
• Probabilistic Distance (PBD):

  $$PBD(A,B)={\displaystyle \frac{\int |P_A-P_B|}{2\int P_{AB}}}$$

  (92)

  $$PBD(S_g,S_t)={\displaystyle \frac{{\sum }_x|f_g\left(x\right)-f_t\left(x\right)|}{2{\sum }_x{f}_g\left(x\right)f_t\left(x\right)}}$$

  (93)

  $$KAP={\displaystyle \frac{f_a-f_c}{N-f_c}}$$

  (94)

  $$f_a=TP+TN$$

  (95)

  $$f_c={\displaystyle \frac{(TN+FN)(TN+FP)+(FP+TP)(FN+TP)}{N}}$$

  (96)
• ROC curve (Receiver Operating Characteristic)

  $$AUC=\sum _{i=1}^{n-1}(FP{R}_{i+1}-FP{R}_i)\times {\displaystyle \frac{TP{R}_i+TP{R}_{i+1}}{2}}$$

  (97)

3.1.6. Spatial Distance Based Metrics

• Distance between crisp volumes:

  $$HD(A,B)=max\left(h\right(A,B),h(B,A\left)\right)$$

  (98)

  $$h(A,B)=\underset{a\in A}{max}\underset{b\in B}{min}||a-b||$$

  (99)
• Average Hausdorff Distance (AVD):

  $$AVD(A,B)=max\left(d\right(A,B),d(B,A\left)\right)$$

  (100)

  $$d(A,B)={\displaystyle \frac{1}{N}}\sum _{a\in A}\underset{b\in B}{min}||a-b||$$

  (101)
• Mahalanobis Distance (MHD):

  $$MHD(x,y)=\sqrt{{(x-y)}^T{S}^{-1}(x-y)}$$

  (102)

  $$MHD(X,Y)=\sqrt{{({\mu }_x-{\mu }_y)}^T{S}^{-1}({\mu }_x-{\mu }_y)}$$

  (103)

  $$S={\displaystyle \frac{n_1+S_1+n_2{S}_2}{n_1+n_2}}$$

  (104)
• Extending the distances to fuzzy volumes

  $$\widehat{H{D}_k}(A,B)={\displaystyle \frac{1}{k}}\sum _{i=1}^{k}H{D}_{{\displaystyle \frac{i}{k}}}(A,B)$$

  (105)

  $$H{D}_{\alpha }(A,B)=HD(A_{\alpha },B_{\alpha })$$

  (106)

  $$\widehat{AV{D}_k}(A,B)={\displaystyle \frac{1}{k}}\sum _{i=1}^{k}AVD(A_{{\displaystyle \frac{i}{k}}},B_{{\displaystyle \frac{i}{k}}})$$

  (107)

  $$\widehat{MH{D}_k}(A,B)={\displaystyle \frac{1}{k}}\sum _{i=1}^{k}MHD(A_{{\displaystyle \frac{i}{k}}},B_{{\displaystyle \frac{i}{k}}})$$

  (108)

3.1.7. Multiple Definition of Met3rics in the Literature

$$VS=2{\displaystyle \frac{|S_t\bigcap S_g|}{|S_t+S_g|}}.100\%$$

(109)

3.2. Evaluation Metrics for Image Segmentation

3.2.1. Error Types of Image Segmentation

• CR (Correct Region):

  $$CR=f(I_g\bigcap I_s)=\sum _{j=1}^M\sum _{i=1}^N|A_j\bigcap B_i|=\sum _{j=1}^M\sum _{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}$$

  (110)
• MD (Miss Detection):

  $$MD=f(I_g\bigcap \widehat{I_s}=\sum _{j=1}^M|A_j|-CR=\sum _{j=1}^M|B_i|-\sum _{j=1}^M\sum _{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}$$

  (111)
• FA (False Alarm):

  $$FA=f(\widehat{I_g}\bigcap I_s)=\sum _{i=1}^N|B_i|-CR=\sum _{i=1}^N|B_i|-\sum _{j=1}^M\sum _{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}$$

  (112)
• BR (Background Region):

  $$BR=f(\widehat{I_g}\bigcap \widehat{I_s}$$

  (113)
• Perfect segmentation:

  $$M=N,and\left\{\begin{array}{cc}{a}_{ji}=0,b_{ji}=|A_i|=|B_i|,c_{ji}=0\hfill & \mathrm{for}\widehat{\delta }(A_j\bigcap B_i)=1\hfill \\ a_{ji}=0,b_{ji}=0,c_{ji}=0\hfill & \mathrm{for}\widehat{\delta }(A_j\bigcap B_i)=0\hfill \end{array}\right.$$
• Completely incorrect segmentation:

  $${\forall }_{i,j}\widehat{\delta }(A_j\bigcap B_i)=0$$

  (114)
• Isolated false alarm

  $$M=N-1>1,and\left\{\begin{array}{cc}i\ne N\hfill & \mathrm{for}\widehat{\delta }(A_j\bigcap B_i)=1\hfill \\ a_{jN}=|A_j|,b_{jN}=0,c_{jN}=|B_N|\hfill & \mathrm{for}\widehat{\delta }(A_j\bigcap B_i)=0\hfill \end{array}\right.$$
• Isolated missed detection:

  $$N=M-1>1,and\left\{\begin{array}{cc}j\ne M\hfill & \mathrm{for}\widehat{\delta }(A_j\bigcap B_i)=1\hfill \\ a_{jM}=|A_M|,b_{Mi}=0,c_{Mi}=|B_i|\hfill & \mathrm{for}\widehat{\delta }(A_j\bigcap B_i)=0\hfill \end{array}\right.$$
• Partial false alarm /miss detection:

  $${\exists }_i\widehat{\delta }(A_i\bigcap B_i)=1and{a}_{ji},b_{ji},c_{ji}\ne 0,N=M-1>1$$

  (115)
• Over segmentation

  $$B{S}_j=\{B_i|\widehat{\delta }(A_i\bigcap B_i)=1,i=1,...,k\},and|A_j|>|B{S}_j|$$

  (116)
• Under segmentation

3.2.2. Evaluation Metrics

• Pixel-based metric:

  $$\begin{array}{cc}\hfill Precision={\displaystyle \frac{CR}{CR+FA}}& ={\displaystyle \frac{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}+{\sum }_{i=1}^N|B_i|-{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}}\hfill \\ \hfill & ={\displaystyle \frac{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{i=1}^N|B_i|}}\hfill \end{array}$$

  (117)

  $$\begin{array}{cc}\hfill Recall={\displaystyle \frac{CR}{CR+MD}}& ={\displaystyle \frac{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}+{\sum }_{j=1}^M|A_j|-{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}}\hfill \\ \hfill & ={\displaystyle \frac{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{i=1}^N|B_i|}}\hfill \end{array}$$

  (118)

  $$F={\displaystyle \frac{2.Precision.Recall}{Precision+Recall}}={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}$$

  (119)
• Object-based metric:Martin’s method

  $$P_{ji}=(1-{\displaystyle \frac{|A_j\bigcap B_i|}{|A_j|}}.|A_j\bigcap B_i|=(1-{\displaystyle \frac{b_{ji}}{a_{ji}+b_{ji}}}).b_{ji}={\displaystyle \frac{a_{ji}+b_{ji}}{a_{ji}+b_{ji}}}$$

  (120)

  $$Q_{ji}=(1-{\displaystyle \frac{|A_j\bigcap B_i|}{|B_i|}}.|A_j\bigcap B_i|=(1-{\displaystyle \frac{b_{ji}}{c_{ji}+b_{ji}}}).b_{ji}={\displaystyle \frac{c_{ji}+b_{ji}}{c_{ji}+b_{ji}}}$$

  (121)

  $$GCE(I_g,I_s)={\displaystyle \frac{1}{n}}min\{\sum _{j=1}^M\sum _{i=1}^N{P}_{ji},\sum _{j=1}^M\sum _{i=1}^N{Q}_{ji}\}$$

  (122)

  where $n={\sum }_{j=1}^M{\sum }_{i=1}^N|A_j\bigcap B_i|$

  $$LCE(I_g,I_s)={\displaystyle \frac{1}{n}}\sum _{j=1}^M\sum _{i=1}^{N}min(P_{ji},Q_{ji})$$

  (123)
• Object-based metric: Polak’s method:

  $$W_{ji}={\displaystyle \frac{\widehat{\delta }(A_j\bigcap B_i)|B_i|}{{\sum }_{K=1}^N\widehat{\delta }(A_j\bigcap B_k)|B_k|}}={\displaystyle \frac{\widehat{\delta }\left(b_{ji}\right(b_{ji}+c_{ji}}{{\sum }_{k=1}^N}}$$

  (124)

3.2.3. Error Measures

Perfect segmentation:

$$\begin{array}{cc}\hfill F={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}& ={\displaystyle \frac{{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}+{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\hfill \\ \hfill & ={\displaystyle \frac{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\hfill \end{array}$$

(125)

Completely inaccurate segmentation

$$F={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^{N}0.b_{ji}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}={\displaystyle \frac{0}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}=0$$

(126)

Isolated false alarm

$$\begin{array}{cc}\hfill F& ={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^{N-1}\widehat{\delta }(A_j\bigcap B_i)b_{ji}+2{\sum }_{j=1}^{M}0·b_{j,N}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\hfill \\ \hfill & ={\displaystyle \frac{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^{N-1}|B_i|}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}={\displaystyle \frac{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|-|B_N|}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\hfill \\ \hfill & =1-{\displaystyle \frac{|B_N|}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}=1-{\displaystyle \frac{|B_N|}{2{\sum }_{i=1}^N|B_i|-|B_N|}}\hfill \\ \hfill & =1-{\displaystyle \frac{|B_N|}{2|B|-|B_N|}}\hfill \end{array}$$

(127)

Isolated miss detection

$$\begin{array}{cc}\hfill F& ={\displaystyle \frac{2{\sum }_{j=1}^{M-1}{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji}+2{\sum }_{i=1}^{N}0·b_{Mi}}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}={\displaystyle \frac{{\sum }_{i=1}^N|B_i|+{\sum }_{j=1}^{M-1}|A_j|}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\hfill \\ \hfill & ={\displaystyle \frac{{\sum }_{i=1}^N|B_i|+{\sum }_{j=1}^M|A_j|-|A_M|}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\hfill \\ \hfill & =1-{\displaystyle \frac{|A_M|}{2{\sum }_{j=1}^M|A_j|-|A_M|}}=1-{\displaystyle \frac{|A_M|}{2|A|-|A_M|}}\hfill \end{array}$$

(128)

Partial false alarm/miss detection

$$F={\displaystyle \frac{2{\sum }_{j=1}^M{\sum }_{i=1}^N\widehat{\delta }(A_j\bigcap B_i)b_{ji})}{{\sum }_{j=1}^M|A_j|+{\sum }_{i=1}^N|B_i|}}\in [0,1]$$

(129)

Over segmentation

$$F={\displaystyle \frac{2{\sum }_{i=1}^k\widehat{\delta }(A_1\bigcap B_i)b_{1i}}{|A_1|+|BS|}}={\displaystyle \frac{2{\sum }_{i=1}^K{b}_{1i}}{|A_1|+|BS|}}={\displaystyle \frac{2|BS|}{|A1|+|BS|}}>{\displaystyle \frac{|BS|}{|A_1|}}={\displaystyle \frac{CR}{CR+MD}}=Recall$$

(130)

Under segmentation

Table 4. Comparison of error measure abilities for three error metrics.

Types	Pixel based	Martin’s metrics	Polak’s metrics
Perfect segmentation	√	√	√
Completely inaccurate segmentation	√	×	×
Isolated false alarm	√	×	×
Isolated missed detection	√	×	×
Partial false alarm/ missed detection	√	×	□
Over-segmentation	√	×	√
Under-segmentation	√	×	√

Table 4. Comparison of error measure abilities for three error metrics.

Types	Pixel based	Martin’s metrics	Polak’s metrics
Perfect segmentation	√	√	√
Completely inaccurate segmentation	√	×	×
Isolated false alarm	√	×	×
Isolated missed detection	√	×	×
Partial false alarm/ missed detection	√	×	□
Over-segmentation	√	×	√
Under-segmentation	√	×	√

4. Evaluating Metrics for Clustering Algorithms

The landscape of clustering algorithms spans classical geometric methods like partition-based K-Means [67] and Mini-Batch K-Means [68], hierarchical approaches such as Agglomerative [69] and BIRCH [70], density-driven techniques including DBSCAN [71] and OPTICS [72], affinity-based models like Affinity Propagation [73] and Mean Shift [74], probabilistic frameworks such as Gaussian Mixture Models [75], and spectral methods like Spectral Clustering [76], alongside modern deep learning paradigms such as Deep Embedded Clustering (DEC) [77], Deep Adaptive Clustering (DAC) [78], Information Maximizing Self-Augmented Training (IMSAT) [79], Variational Deep Embedding (VaDE) [80], and the recently proposed EDCWRN [81] that integrates representation weighting and neighborhood relationships. Evaluating these diverse methodologies necessitates metrics tailored to data characteristics: traditional criteria (e.g., inertia, silhouette scores) assume Euclidean structures, while emerging challenges—such as mixed interval-categorical data or non-Euclidean distances—demand specialized validation frameworks. This section reviews classical and recently developed metrics, emphasizing their applicability to both conventional and deep clustering architectures. To rigorously define these metrics, we first formalize the foundational elements of clustering [82]:

Dataset Definition. Let X denote a dataset containing N objects, where each object is represented as a vector in an F-dimensional space:

$$X=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N\},X\subseteq {\mathbb{R}}^F.$$

Clustering Partition. A partition (or clustering) of X divides the dataset into K distinct, non-overlapping clusters:

$$\mathcal{C}=\{C_1,C_2,\dots ,C_K\},$$

satisfying:

• Coverage: ${\bigcup }_{k=1}^K{C}_k=X$,
• Disjointness: $C_k\cap C_l=\varnothing \forall k\ne l$.

Cluster and Dataset Centroids.

• The cluster centroid (mean vector) for $C_k$:

  $${\mu }_k={\displaystyle \frac{1}{|C_k|}}\sum _{{\mathbf{x}}_i\in C_k}{\mathbf{x}}_i$$
• The dataset centroid (global mean):

  $$\mu ={\displaystyle \frac{1}{N}}\sum _{{\mathbf{x}}_i\in X}{\mathbf{x}}_i$$

4.1. Cluster Validity Indices (CVIs)

The evaluation of clustering results is a critical aspect of the clustering process, often achieved using cluster validity indices (CVIs) that assess the quality of partitions based on various criteria such as compactness and separation. While several comparative studies of CVIs exist, including the extensive analysis by Arbelaitz et al.,arbelaitz2013extensive, which systematically evaluated 30 indices across diverse configurations, this section provides a structured overview of these indices. By organizing their definitions and key properties in a tabular format, we aim to offer a clear and accessible reference for understanding the characteristics and applicability of each CVI in different clustering scenarios.

Table 5. This is a table caption. Tables should be placed in the main text near to the first time they are cited.

Index Name	Description/Definition	Citation
Dunn Index (D↑)	Measures the ratio of the minimum inter-cluster distance to the maximum intra-cluster distance. Higher values indicate better clustering.	[83]
Calinski-Harabasz Index (CH↑)	Ratio of the sum of between-cluster dispersion to within-cluster dispersion. Higher values indicate better clustering.	[84]
Davies-Bouldin Index (DB↓)	Measures the average similarity between each cluster and the cluster most similar to it. Lower values indicate better clustering.	[85]
Silhouette Index (Sil↑)	Combines cohesion (average intra-cluster distance) and separation (average nearest-cluster distance) into a single measure. Values range from -1 to 1, where higher values indicate better clustering.	[86]
Gamma Index (G↓)	Measures the correlation between the distances of pairs of points and their clustering assignments. Higher values indicate better clustering.	[87]
C Index (CI↓)	Normalized measure of intra-cluster distances compared to minimum and maximum possible distances. Lower values indicate better clustering.	[88]
S_Dbw Index (SDbw↓)	Combines intra-cluster variance and inter-cluster density to evaluate clustering quality. Lower values indicate better clustering.	[89]
CS Index (CS↓)	Estimates cohesion by cluster diameters and separation by the nearest neighbor distance. Lower values indicate better clustering.	[90]
Symmetry Index (Sym↑)	Measures the symmetry of clusters around their centroids using the Point Symmetry Distance. Lower values indicate better clustering.	[91]
COP Index (COP↓)	Combines intra-cluster cohesion (distance to centroid) and inter-cluster separation (furthest neighbor distance). Lower values indicate better clustering.	[92]
Negentropy Increment (NI↓)	Based on entropy, measures the increase in negentropy when clustering is applied. Higher values indicate better clustering.	[93]
Score Function (SF↑)	Combines between-cluster dispersion and within-cluster dispersion using a weighted exponential function. Higher values indicate better clustering.	[94]
Generalized Dunn Index (gD31, gD41, gD51, gD33, gD43, gD53)	Variants of the Dunn index using different measures for cohesion and separation. Higher values indicate better clustering.	[95]
Graph theory based Dunn and Davies– Bouldin variations	Dunn index variants that use graph-based measures (Minimum Spanning Tree, Relative Neighborhood Graph, Gabriel Graph) for cohesion.	[96]
Point Symmetry-Based Indices (SymDB↓, SymD↑, Sym33↑)	Variants of the Davies-Bouldin, Dunn, and Generalized Dunn indices using Point Symmetry Distance for cohesion. Lower values indicate better clustering.	[97]
Davies–Bouldinn* (DB*↓)	A modification of the Davies-Bouldin index.	[98]
SV Index (SV↑)	Combines separation (nearest neighbor distance) and cohesion (distance of border points to centroid). Lower values indicate better clustering.	[99]
OS Index (OS↑)	Combines a more complex separation estimator with cohesion (distance of border points to centroid). Lower values indicate better clustering.	[99]

Table 5. This is a table caption. Tables should be placed in the main text near to the first time they are cited.

Index Name	Description/Definition	Citation
Dunn Index (D↑)	Measures the ratio of the minimum inter-cluster distance to the maximum intra-cluster distance. Higher values indicate better clustering.	[83]
Calinski-Harabasz Index (CH↑)	Ratio of the sum of between-cluster dispersion to within-cluster dispersion. Higher values indicate better clustering.	[84]
Davies-Bouldin Index (DB↓)	Measures the average similarity between each cluster and the cluster most similar to it. Lower values indicate better clustering.	[85]
Silhouette Index (Sil↑)	Combines cohesion (average intra-cluster distance) and separation (average nearest-cluster distance) into a single measure. Values range from -1 to 1, where higher values indicate better clustering.	[86]
Gamma Index (G↓)	Measures the correlation between the distances of pairs of points and their clustering assignments. Higher values indicate better clustering.	[87]
C Index (CI↓)	Normalized measure of intra-cluster distances compared to minimum and maximum possible distances. Lower values indicate better clustering.	[88]
S_Dbw Index (SDbw↓)	Combines intra-cluster variance and inter-cluster density to evaluate clustering quality. Lower values indicate better clustering.	[89]
CS Index (CS↓)	Estimates cohesion by cluster diameters and separation by the nearest neighbor distance. Lower values indicate better clustering.	[90]
Symmetry Index (Sym↑)	Measures the symmetry of clusters around their centroids using the Point Symmetry Distance. Lower values indicate better clustering.	[91]
COP Index (COP↓)	Combines intra-cluster cohesion (distance to centroid) and inter-cluster separation (furthest neighbor distance). Lower values indicate better clustering.	[92]
Negentropy Increment (NI↓)	Based on entropy, measures the increase in negentropy when clustering is applied. Higher values indicate better clustering.	[93]
Score Function (SF↑)	Combines between-cluster dispersion and within-cluster dispersion using a weighted exponential function. Higher values indicate better clustering.	[94]
Generalized Dunn Index (gD31, gD41, gD51, gD33, gD43, gD53)	Variants of the Dunn index using different measures for cohesion and separation. Higher values indicate better clustering.	[95]
Graph theory based Dunn and Davies– Bouldin variations	Dunn index variants that use graph-based measures (Minimum Spanning Tree, Relative Neighborhood Graph, Gabriel Graph) for cohesion.	[96]
Point Symmetry-Based Indices (SymDB↓, SymD↑, Sym33↑)	Variants of the Davies-Bouldin, Dunn, and Generalized Dunn indices using Point Symmetry Distance for cohesion. Lower values indicate better clustering.	[97]
Davies–Bouldinn* (DB*↓)	A modification of the Davies-Bouldin index.	[98]
SV Index (SV↑)	Combines separation (nearest neighbor distance) and cohesion (distance of border points to centroid). Lower values indicate better clustering.	[99]
OS Index (OS↑)	Combines a more complex separation estimator with cohesion (distance of border points to centroid). Lower values indicate better clustering.	[99]

4.2. Internal Validation Methods

Internal validation methods are used to assess the quality of clustering results without relying on external information, such as ground truth labels. These methods evaluate the clustering structure based solely on the input data, making them particularly useful in unsupervised learning tasks.

Internal validation metrics are particularly versatile and are commonly applied to two major families of clustering algorithms: partitional clustering algorithms (e.g., k-means, k-medoids) and hierarchical clustering algorithms (e.g., agglomerative or divisive methods) [100]. Below, we delve into the specifics of these metrics and their applications to these algorithmic families.

4.2.1. Metrics for Partitional Clustering Algorithms

Partitional clustering algorithms divide the data into non-overlapping groups. Internal validation metrics for these algorithms are typically based on proximity measures (e.g., distance or similarity) and aim to evaluate how well the clusters are formed. Some commonly used metrics include:

• Sum of Squared Errors (SSE): Measures the compactness of clusters by calculating the sum of squared distances between each data point and its cluster centroid. Lower SSE values indicate better clustering.
• Silhouette Coefficient: Combines cohesion and separation into a single metric. It compares the average distance of a point to other points in its cluster (cohesion) with the average distance to points in the nearest neighboring cluster (separation). Values range from -1 to 1, where higher values indicate better clustering.
• Calinski-Harabasz Index (CH): Also known as the variance ratio criterion, this index measures the ratio of between-cluster dispersion to within-cluster dispersion. Higher CH values indicate better-defined clusters.
• Dunn Index: Evaluates the ratio of the smallest distance between clusters (separation) to the largest intra-cluster distance (cohesion). A higher Dunn Index signifies better clustering.
• Xie-Beni Index: Originally designed for fuzzy clustering, this metric estimates the compactness and separation of clusters. It is also applicable to hard clustering methods.

4.2.2. Metrics for Hierarchical Clustering Algorithms

Hierarchical clustering algorithms produce a tree-like structure (dendrogram) that represents nested groupings of data. Internal validation metrics for these algorithms often evaluate the quality of the dendrogram itself. Key metrics include:

• Cophenetic Correlation Coefficient (CPCC): Measures how faithfully the dendrogram preserves the pairwise distances between data points. High CPCC values indicate that the hierarchical clustering algorithm has captured the underlying structure of the data.
• Hubert Statistic: Similar to CPCC, this metric evaluates the concordance between the proximity matrix and the cophenetic matrix derived from the dendrogram. Higher Hubert values suggest better clustering.
• Proximity Matrix Correlation: Compares the actual proximity matrix to an idealized block-diagonal matrix based on the clustering results. High correlation indicates well-formed clusters.

4.2.3. Innovative Perspective on Internal Validation

While traditional internal validation metrics like SSE and Silhouette Coefficient are widely used, they often align closely with the optimization objectives of specific clustering algorithms (e.g., k-means minimizes SSE). This alignment can introduce bias, making it challenging to compare different algorithms fairly. To address this limitation, we propose a novel approach that combines multiple metrics into a unified framework, enabling a more holistic evaluation of clustering results. By integrating cohesion, separation, and proximity measures, our method provides a comprehensive assessment of clustering quality, independent of the algorithm used.

Furthermore, we emphasize the importance of computational efficiency in large-scale clustering tasks. Metrics like the Silhouette Coefficient, with a complexity of $O\left(n^2\right)$, may become impractical for large datasets. To overcome this, we advocate for the development of approximate metrics that retain the interpretability of traditional indices while significantly reducing computational costs.

4.2.4. Conclusion

Internal validation methods play a pivotal role in unsupervised learning by enabling researchers to evaluate clustering results without relying on external labels. By focusing on cohesion and separation, these methods provide valuable insights into the quality of clustering structures. However, as clustering tasks grow in complexity and scale, there is a pressing need for innovative metrics that balance interpretability, fairness, and computational efficiency. We hope that the framework presented here inspires future research and establishes a robust foundation for evaluating clustering algorithms.

4.2.5. Partitional Methods

Partition clustering assigns a set of data to k clusters using repeated processes. In these processes, n data are classified into k-clusters. The predefined criterion function J assigns the data to the kth numerical set according to the calculation of maximization and minimization in k sets. The internal validation value of a set of K clusters can be decomposed as the sum of the validation values for each cluster as follows [101]:

$$genera{l}_{validity}=\sum _{i=1}^K{w}_{i}validity\left(C_i\right)$$

(131)

• cohesion:Cohesion evaluates how close the elements of a cluster are to each other. This index is defined as follows [100]:

  $$cohesion\left(C_i\right)=\sum _{x\in C_i,y\in C_i}proximity(x,y)$$

  (132)
• Separation:eparation measures the level of separation between clusters. This index is defined as follows [100]:

  $$separation(C_i,C_j)=\sum _{x\in C_i,y\in C_j}proximity(x,y)$$

  (133)
• Sum of Squared Errors Within Cluster:It should be noted that the coherence metric defined above is equivalent to the cluster Sum of Squared Errors Within Cluster(SSW). In general, this metric can be defined as follows [101]:

  $$SSE\left(C_i\right)=\sum _{x\in C_i}d{(c_i,x)}^2={\displaystyle \frac{1}{2m_i}}\sum _{x\in C_i}\sum _{y\in C_i}d{(x,y)}^2$$

  (134)
• Maximize distance between clusters: This metric maximizes the distance between clusters and is defined as follows:

  $$SSB=\sum _{i=1}^K{m}_{i}d{(c_i,c)}^2={\displaystyle \frac{1}{2K}}\sum _{i=1}^k\sum _{j=1}^k{\displaystyle \frac{m}{K}}d{(c_i,c_j)}^2$$

  (135)
• Calisnki-Harabasz coefficient(CH): The CH, makes the final decision based on the measure of intra-cluster dispersion and inter-cluster dispersion. The goal is to choose the number of clusters that maximize CH. This criterion is known as the variance ratio criterion and is defined as follows [84]:

  $$CH={\displaystyle \frac{\frac{SS{B}_m}{(M-1)}}{\frac{SS{E}_M}{\left(M\right)}}}$$

  (136)
• Dunn index: This index has been reviewed before and can also be defined as follows:

  $$D=\underset{1<i<k}{min}\left\{\underset{1<j<k,i\ne j}{min}\left\{{\displaystyle \frac{\delta (C_i,C_j)}{{max}_{1<l<K}\{\Delta \left(C_l\right)\}}}\right\}\right\}$$

  (137)

  $$\Delta \left(C_i\right)=\underset{x,y\in c_i}{max}\left\{d(x,y)\right\}$$

  (138)

  $$\delta (C_i,C_j)=\underset{x\in C_i,y\in C_j}{min}\left\{d(x,y)\right\}$$

  (139)
• Xie-Beni score:This index is ratio-based, and its numerator estimates the level of data density in the same cluster, and its denominator estimates the level of separation of data from different clusters. It is designed for fuzzy clustering, but it can also be used for hard clustering. This index can be defined as follows [102]:

  $$XB={\displaystyle \frac{{\sum }_{i=1}^M{\sum }_{k=1}^M{u}_{ik}^2|\parallel x_i,C_k{\parallel |}^2}{N_{t\notin s}min\{\parallel |C_t-C_s|\parallel \}}}$$

  (140)
• Ball-Hall index: This index is a dispersion measure that performs clustering based on the quadratic distance of the cluster points concerning their center and is defined as follows [103]:

  $$BH={\displaystyle \frac{SS{E}_M}{M}}$$

  (141)
• Hartigan index: This index performs clustering based on the logarithmic relationship between the sum of squares within the cluster and the sum of squares between the clusters and is defined as follows [104]:

  $$H=log\left({\displaystyle \frac{SS{B}_M}{SS{E}_M}}\right)$$

  (142)
• Xu coefficient: This coefficient considers the clustering of dimensions D of data, the number N of data samples, and the sum of squared errors of $SSEM$ from M clusters [105]:

  $$Xu=D{log}_2\left(\sqrt{{\displaystyle \frac{SS{E}_M}{D{N}^2}}}\right)+logM$$

  (143)
• silhouette coefficientThis index has been reviewed before and can also be defined as follows [100]:

  $$a\left(i\right)={\displaystyle \frac{1}{|C_a|}}\sum _{j\in C_a,i\notin j}d(i,j)$$

  (144)

  $$b\left(i\right)=\underset{C_b\notin C_a}{min}{\displaystyle \frac{1}{|C_b|}}\sum _{j\in C_b}d(i,j)$$

  (145)

  $$s\left(i\right)={\displaystyle \frac{b\left(i\right)-a\left(i\right)}{max\left\{a\right(i),b(i\left)\right\}}}$$

  (146)

  $$S={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}s\left(i\right)$$

  (147)

4.2.6. Hierarchical Methods

This clustering is a step-by-step method of creating a hierarchy of clusters, which can be defined as agglomerative (bottom-up approach) and divisive (top-down approach). In agglomerative mode, the clustering algorithm starts by considering each data point as a separate cluster. So, if there are N data points, it starts with N clusters. At each step, the algorithm merges the two clusters that are closest to each other until all the clusters are merged into one big cluster containing all the data points. In the divisive approach, it starts with all the data points in one cluster. At each step, the algorithm divides the cluster until each cluster contains only one data point. This method is less common and more computationally intensive compared to cumulative clustering [106]. Several internal validation techniques have also been proposed and tested with hierarchical clustering algorithms, which are discussed below.

• Cophenetic Correlation Coefficient (CPCC): A hierarchical clustering algorithm is used to evaluate the results. This correlation coefficient was proposed as the correlation between the cophenetic matrix $P_c$, containing cophenetic distances, and the proximity matrix P, containing similarities. This correlation coefficient is defined as [107].

  $$CPCC={\displaystyle \frac{{\sum }_{i<j}(d_{i,j}-\widehat{d})(d_{i,j}^{\ast }-\widehat{d^{\ast }})}{\sqrt{{\sum }_{i<j}{(d_{i,j}-\widehat{d})}^2{\sum }_{i<j}(d_{i,j}^{\ast }-{\widehat{d^{\ast }}}^2}}}$$

  (148)

  $$\widehat{d}={\displaystyle \frac{{\sum }_{i<j}{d}_{ij}}{2(n^2-n)}}$$

  (149)

  $$\widehat{d^{\ast }}=\sqrt{{\displaystyle \frac{{\sum }_{i<j}{(d_{i,j}-\widehat{d})}^2}{{\sum }_{i<j}{\left(d_{ij}^{\ast }\right)}^2}}}$$

  (150)
• Hubert Statistic: Let $S_{+}$ and $S_{-}$ be the number of congruent and discordant pairs, respectively. This index is defined as [100]:

  $$\gamma ={\displaystyle \frac{S_{+}-S_{-}}{S_{+}+S_{-}}}$$

  (151)

4.3. External Validation Met3hods

These validation methods are usually defined in the form of supervised learning. These methods are performed by including additional information in the clustering validation process, such as external class labels for the training samples. Since unsupervised learning techniques are generally used when such information is not available, external validation methods are not used in most clustering problems. However, they can be applied when external information is available as well as when generating synthetic data from a real data set [108]. These evaluation criteria are divided into three subsections:

4.3.1. Matching Sets

Some methods identify the relationship between each cluster identified in C and its natural correspondence to the resulting reference classes defined by P.

• Precision: Counts the true positives.

  $$Pr={\displaystyle \frac{TP}{TP+FP}}$$

  (152)
• Recall: Calculates the percentage of elements that are correctly included in a cluster:

  $$R={\displaystyle \frac{TP}{TP+FN}}$$

  (153)
• F-Measure: combines precision and recall in a single metric.

  $$F_a={\displaystyle \frac{2\ast Pr\ast R}{Pr+R}}$$

  (154)
• Purity: It is used to evaluate whether each cluster contains only instances of the same class.

  $$U=\sum _i{p}_i(\underset{j}{max}{\displaystyle \frac{p_{ij}}{p_i}}$$

  (155)
  Where $pi={\displaystyle \frac{n_i}{n}}$ , $p_j={\displaystyle \frac{n_j}{n}}$ and $p_{ij}={\displaystyle \frac{n_{ij}}{n}}$

4.3.2. Peer-to-Peer Correlation

• Jaccard coefficient:

  $$J={\displaystyle \frac{TP}{TP+FP+FN}}$$

  (156)

  $$J={\displaystyle \frac{{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{n_{ij}}{2}\right)}{{\sum }_i\left(\genfrac{}{}{0pt}{}{n_i}{2}\right)+{\sum }_j\left(\genfrac{}{}{0pt}{}{n_j}{2}\right)-{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{n_{ij}}{2}\right)}}$$

  (157)
• Rand coefficient:

  $$Rand={\displaystyle \frac{TP+TN}{M}}$$

  (158)

  $$Rand={\displaystyle \frac{\left(\genfrac{}{}{0pt}{}{n}{2}\right)-{\sum }_i\left(\genfrac{}{}{0pt}{}{n_i}{2}\right)+{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{n_{ij}}{2}\right)-\left(\genfrac{}{}{0pt}{}{n}{2}\right)}{\left(\genfrac{}{}{0pt}{}{n}{2}\right)}}$$

  (159)
• Folkes and Mallows coefficient:

  $$FM=\sqrt{{\displaystyle \frac{TP}{TP+FP}}+{\displaystyle \frac{TP}{TP+FN}}}$$

  (160)

  $$FM=={\displaystyle \frac{{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{n_{ij}}{2}\right)}{\sqrt{{\sum }_i\left(\genfrac{}{}{0pt}{}{n_i}{2}\right)\ast {\sum }_j\left(\genfrac{}{}{0pt}{}{n_j}{2}\right)}}}$$

  (161)
• Hubert statistical coefficient:

  $$\Gamma ={\displaystyle \frac{1}{M}}\sum _{i=1}^{n-1}\sum _{j=i+1}^n{X}_{ij}{Y}_{ij}$$

  (162)

  $$\Gamma ={\displaystyle \frac{\left(\genfrac{}{}{0pt}{}{n}{2}\right){\sum }_{ij}\left(\genfrac{}{}{0pt}{}{n_{i}j}{2}\right)-{\sum }_i\left(\genfrac{}{}{0pt}{}{n_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{n_j}{2}\right)}{\sqrt{{\sum }_i\left(\genfrac{}{}{0pt}{}{n_i}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{n_j}{2}\right)[\left(\genfrac{}{}{0pt}{}{n}{2}\right)-{\sum }_i\left(\genfrac{}{}{0pt}{}{n_i}{2}\right)][\left(\genfrac{}{}{0pt}{}{n}{2}\right)-{\sum }_j\left(\genfrac{}{}{0pt}{}{n_j}{2}\right)]}}}$$

  (163)

4.3.3. Measures Based on Information Theory

• Entropy:

  $$H=-\sum _i{p}_i(\sum _i{\displaystyle \frac{p_{ij}}{p_i}}log{\displaystyle \frac{p_{ij}}{p_i}}$$

  (164)
• Mutual information:

  $$MI=\sum _i\sum _j{p}_{ij}log{\displaystyle \frac{p_{ij}}{p_i{p}_j}}$$

  (165)

  where $p_{ij}=n_{ij}/n$,$p_i=n_i/n$, and $p_j=n_j/n$.

5. Signal Mining

5.1. Similarity and Distance Met3rics

In this section, similarity and distance criteria are introduced. These criteria are used on two or more signals. Let us assume that we have two sequences, $P=p\left(k\right)$, $k=1,2,\dots ,K$, and $Q=q\left(k\right)$, $k=1,2,\dots ,K$, where $p\left(k\right)$ and $q\left(k\right)$ are the observed values of P and Q at time k, respectively. A variety of typical metrics, that are potentially available for EEG analysing, are introduced to measure the similarity and distance between P and Q.

5.1.1. Euclidean Distance (ED)

The ED between P and Q calculated as:

$$d^{ED}=\sqrt{\sum _{k=1}^n{((p\left(k\right)-q\left(k\right))}^2}$$

(166)

The ED similarity calculated as follow:

$$S^{ED}={\displaystyle \frac{1}{d^{ED}}}$$

(167)

5.1.2. Pearson Correlation Coefficient Distance (PCCD)

The PCCD between P and Q can be calculated by

$$d^{PCCD}={\displaystyle \frac{{\sum }_{k=1}^k(p\left(k\right)-\widehat{p})(q\left(k\right)-\widehat{q}}{\sqrt{{\sum }_{k=1}^k(p\left(k\right)-\widehat{p})}\sqrt{{\sum }_{k=1}^k(q\left(k\right)-\widehat{q}}}}$$

(168)

So, the similarity defined by PCCD is then calculated by

$$S^{PCCD}=|d^{PCCD}|$$

(169)

5.1.3. Symmetric Kullback–Leibler Divergence (SKLD)

The SKLD between P and Q can be calculated by:

$$D(P||Q)=\sum _{k=1}^K\left(P\left(k\right)log\left({\displaystyle \frac{p\left(k\right)}{q\left(k\right)}}\right)\right)$$

(170)

$$D(Q||P)=\sum _{k=1}^K\left(q\left(k\right)log\left({\displaystyle \frac{q\left(k\right)}{p\left(k\right)}}\right)\right)$$

(171)

Kullback–Leibler distance metrics [35] is calculated by:

$$d^{SKLD}={\displaystyle \frac{D(P||Q)+D(Q||P)}{2}}$$

(172)

Then, the similarity can be gotten as:

$$S^{SKLD}={\displaystyle \frac{1}{d^{SKLD}}}$$

(173)

5.1.4. Hellinger Distance (HD)

The HD between P and Q can be calculated by

$$d^{HD}={\displaystyle \frac{1}{\sqrt{2}}}||\sqrt{P}-\sqrt{Q}{||}_2$$

(174)

5.1.5. Bhattacharyya Distance (BD)

BD which was proposed by Bhattacharyya in [40], also known as the Hellinger. The BD between P and Q is defined as:

$$s^{BD}=-ln\left(BC(P,Q)\right)$$

(175)

where $BC(X,Y)$ is the Bhattacharyya coefficient.

$$BC(P,Q)=\sum _{k=1}^K\sqrt{p\left(k\right)q\left(k\right)}$$

(176)

5.1.6. Minkowski

In general, the distance $d_{ij}$ between any two points in n-dimensional space may be calculated by the equation given by Minkowski:

$$d_{ij}=[\sum _{i=k}^n{|p\left(k\right)-q\left(k\right)|}^{PP}{]}^{{\displaystyle \frac{1}{PP}}}$$

(177)

$PP$ determining the type of distance:

• $PP=1$: city block distance, or Manhattan distance.
• $PP=1$: In binary data, Hamming distance.
• $PP=2$: well-known Euclidean distance.

5.1.7. Entropy

Entropy can be used to measure the difference between two probability distributions. Considering two different probability distributions P and Q, their relative entropy can be described in the following form:

$$D_{EN}(P||Q)=\sum _{x=1}^{r}P\left(x\right).lo{g}_2{\displaystyle \frac{P\left(x\right)}{Q\left(x\right)}}$$

(178)

5.1.8. Kullback-Leibler

In order to make the relative entropy able to satisfy the definition of the distance, the redefined formula is rewritten as[REF ]:

$$RD(P||Q)={\displaystyle \frac{D_{kl}(P||Q)+D_{KL}(Q||P)}{2}}$$

(179)

From the above analysis, there is:

$$\begin{array}{c}RD(P||Q)=\left\{{\displaystyle \frac{{\sum }_{x=1}^{r}P\left(x\right).lo{g}_2\left[\frac{P\left(x\right)}{Q\left(x\right)}\right]+{\sum }_{x=1}^{r}Q\left(x\right).lo{g}_2\left[\frac{Q\left(x\right)}{P\left(x\right)}\right]}{2}}\right\}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\displaystyle \frac{{\sum }_{x=1}^r[P\left(x\right)-Q\left(x\right).lo{g}_2\left[\frac{P\left(x\right)}{Q\left(x\right)}\right]}{2}}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\displaystyle \frac{{\sum }_{x=1}^r[Q\left(x\right)-P\left(x\right)].lo{g}_2\left[\frac{Q\left(x\right)}{P\left(x\right)}\right]}{2}}=RD(Q||P)\end{array}$$

(180)

5.1.9. Angle

The angle between two vectors can be calculated using the following formula:

$$\theta =acos\left({\displaystyle \frac{\overrightarrow{P}.\overrightarrow{Q}}{|\overrightarrow{P}|.|\overrightarrow{Q}|}}\right)$$

(181)

where $\theta$ represents the angle between the vectors, $\overrightarrow{u}$ and $\overrightarrow{v}$ are the two vectors, $\overrightarrow{u}·\overrightarrow{v}$ is the dot product of the vectors, and $\overrightarrow{P}$ and $\overrightarrow{Q}$ denote the magnitudes of the vectors $|\overrightarrow{P}|$ and $|\overrightarrow{Q}|$ respectively.

5.1.10. Itakura-Saito

The Itakura–Saito distance (or Itakura–Saito divergence) is a measure of the difference between an original spectrum and an approximation of that spectrum. Although it is not a perceptual measure, it is intended to reflect perceptual (dis)similarity. It was proposed by Fumitada Itakura and Shuzo Saito in the 1960s while they were with NTT.

$$Dist(P,Q)={\displaystyle \frac{1}{2\mathrm{\Pi }}}{\int }_{-\mathrm{\Pi }}^{\mathrm{\Pi }}[{\displaystyle \frac{P}{Q}}-log{\displaystyle \frac{P}{Q}}-1]dw$$

(182)

5.1.11. Dice, Cosine, and Jaccard Coefficient [Reff ]

$$DiceCoefficient=2{\displaystyle \frac{|P\bigcap Q|}{|P|+|Q|}}$$

(183)

$$CosineCoefficient=2{\displaystyle \frac{|P\bigcap Q|}{{|P|}^{\frac{1}{2}}+{|Q|}^{\frac{1}{2}}}}$$

(184)

$$JaccardCoefficient=2{\displaystyle \frac{|P\bigcap Q|}{|P|+|Q|-|P\bigcap Q|}}$$

(185)

6. Conclusions

Machine learning and deep learning are powerful tools in learning model research and are becoming increasingly pervasive. The adoption of these systems is expanding, accelerating the shift towards a more algorithmic society, meaning that algorithm-based decisions have greater potential for significant societal impact. However, most of these sophisticated decision support systems remain complex black boxes, meaning that their internal logic and inner workings are hidden from the user, and even experts cannot fully understand the logic behind their predictions. In addition, new regulations and highly regulated fields have mandated auditability and verifiability of decisions, increasing the demand for the ability to question, understand, and trust machine learning systems, for which interpretability is essential. Indeed, care must be taken when choosing metrics to evaluate model outputs and interpret results. There is no single metric that fits all. Several appropriate performance metrics should be selected based on an understanding of the dataset and the research question. Reproducibility of results is crucial for readers to trust the reported metrics, as is a discussion of potential biases in the data and model that could affect the metrics. After reporting model performance, biases should be considered. This paper extensively reviews evaluation metrics appropriate for machine learning and deep learning tasks such as classification, clustering, and ranking. Our main contribution is to systematically categorize and discuss these evaluation metrics based on the learning problem and their application context. Unlike previous reviews, we conducted a broad survey. Choosing appropriate evaluation metrics is crucial for the effective evaluation of machine learning models. Different metrics provide distinct insights depending on the problem type, data characteristics, and application goals. Hence, we attempted to review nearly 250 different evaluation metrics in different categories, including text mining, image mining, clustering, and signal mining. Future research could be devoted to evaluating criteria in large language models. This area of research has devoted itself to different criteria for interpretability.