GS-OCMXM: Towards Parallel and GPU Accelerated Computing on GS-Based Multi-Cross-Modality Learning for Accurate Segmentation of Ultrasonography Thyroid Nodule Cancer Detection with Limited Labels

1. Introduction

The early and accurate diagnosis of computer-aided thyroid cancer has been the subject of research over the past few decades [1]. Ultrasonography remains a crucial tool in the workup of thyroid nodules. The integration of multi-modal ultrasonography techniques enhances the evaluation and management, allowing for more informed clinical decisions [2]. Automatic medical image segmentation plays a critical role in scientific research and medical care [3]. Timely detection of thyroid nodule ultrasound could provide a tool for decreasing both cancer mortality and incidence [4]. The operative clinical trials link on TN assessment staging system was used to stratify the presence/absence and risk level of the lesions. Detection of thyroid nodules’ lesions would allow surveillance to be performed, making early detection of the transformation to cancer possible [5]. However, despite significant advances, automatic TN detection in real time is still an unsolved problem. One concept for predicting the thyroid nodule lesions is to construct statistical models using some lesion appearance features or sonographic characteristics [6]. A more comprehensive analysis using more quantitative imaging features may provide better accuracy in predicting thyroid nodules [7]. For this reason, we investigated the feasibility of ultrasonography in predicting thyroid nodules.

Before proposing our models, we had established a number of challenges that needed to be overcome in order to make the developed AI models better applicable to clinical practice.

The first challenge is about difficult for fine grained US-guided thyroid nodule detection with such US-TN images themselves such as having multi-classes [8], shape variations [9], indistinct boundaries [10], appearance, the heterogeneous and dynamic environment compared to other patterns on MRI, CT or X-ray in 2D and etc [11]. and we need to combine distant view and close-up view too better achieve target lesion detection [12], we call it global and local view in this thesis. Also because of large variety of TN types and characteristics. What is more, some very subtle appearance in US-TN, which is hard to be detected [13]. For the above reasons, we need highly customized or a new CNN design to deal with. Meanwhile, for a training model, a good quality and large quantity of training data is also important, it requires high resolution and take a data augmentation to achieve high performance training models [14]. And on the hardware side, we need to extract effective features and different model’s fusion [15]. More importantly, the last one is about the domain adaptation, and DA for short, which has recently raised strong interests in the medical imaging community [16]. By encouraging algorithms to be robust to unseen situations or different input data domains [17]. Domain Adaptation improves the applicability of machine learning approaches to various clinical settings [18]. While a large variety of DA techniques has been proposed, most of these techniques have been validated either on private datasets or on small publicly available datasets [19]. Moreover, these datasets mostly address single-class problems [20].

• Our research targets and goals:

  (a)

  Our motivation is to be able to detect, identify, locate and mark the boundary of the target lesions for fine grained TN detection.

  (b)

  The thyroid nodules system should be an adaptive, robust, and fast prediction system that works for US-TN image data with multi-modal technology [21].

  (c)

  The system can be able to work with real-time ultrasonography imaging.

  (d)

  Future more, in order to achieve domain adaptation(DA), and It will be able to scale and fuse with features and data from other sources [22] (e.g. different thyroid nodules features from various devices.)

  (e)

  And algorithm wise, we will implement a multi-level optimized adaptive precise segmentation approach for thyroid nodules. This will involve comparatively testing various state-of-the-art deep learning algorithms available in open source. We aim to improve and extend these models to ultimately design a deep learning model that is most suitable for our specific thyroid nodule case.

  (f)

  And it will test under non-public real-life US-TN images and video clips.

  (g)

  In last, we construct it to an operational prototype.

• In summary, the major contributions of this paper are as follows.

  (a)

  In our approach, we present an efficient 4-level framework including DuS-KFCM, Patch Learning, Cross Learning and the lowest layer called parallel computing for the US-TN image segmentation. And they are a progressive relationship.

  (a)

  We proposed a feature learning scheme that engineers class-specific features that are generically discriminative.

  (b)

  What’s more, the developed novel multiple level one class cross learning model(ML-OCX for short) considers different contributions of patches to predict the final image-level by introducing GS agent automatically.

  (c)

  We construct a novel early gastric image dataset which consist of 4 types of US-TN images. The extensive experiments on the dataset demonstrate the effectiveness of our ML-OCX model and our method outperforms other learning methods.

2. Overview of the Proposed Model

The overall procedure of the ML-OCX model can be divided into 4 part, which are mainly the physical layer and 3 upper model architecture respectively. Here the brief introduction of these 4 parts is given.

2.1. The Overall Procedure of The Layer Architecture

In this section, we first introduce the proposed multi-level one class cross learning model with parallel GPU computing, based on parallel computing structure, global Dus-KFCM fuzzy model, local patch learning model and one class cross learning block, a novel multi-level layer fusion network of ML-OCX model is designed, as shown in Figure 1. The following will elaborate on these 4-levels aspects.

2.2. Methodology and Design

2.2.1. Dus-KFCM Fuzzy Segmentation

In our early work, we the continuous interactions and selective focus are used to better capture the information. The GLCM feature extraction method is employed, which consists of a 22 spatial attention block. We fuse the multi attention block and 3 channel-level attention blocks to improve the SKFCM model [23]. Specifically, the GLCM feature map can locate the informative regions, can channel-level attention block aim to enhance the feature presentation by modeling the interdependence between convolution channels.

2.2.2. Local Patch Learning

The local patch learning is the higher level, which are designed to improve the segmentation result of global fuzzy clustering model. The model divides into two stages, including LPM and GPM model separately [24], and the first stage is rough segmentation model to utilize prior knowledge of one modality to enhance the segmentation performance of another modality. And we input different kinds of the raw gastric cancer images firstly, we take a global pre-segmentation using the GLCM. For each image, we could get a ‘stream-patch’ from a set of image patches to a vector [25]. And we present a method to analyze patches by embedding them to a vector space [26]. The cross-correlation with patches analysis in calculated for each feature vectorized representations are fused as the super-vector and normalized [27]. We could get the significant patches based on the trained optimized parameters. For different type of patch of different class of image, including lesion type1t̃ype4, we divide different class of the patch to the different GPM model, we could update the GPM model by optimizing parameters [28]. And then we could transfer the GPM output to the LPM model. we use the fuzzy segmentation result as the new gastric image set. And different class of patches go to different LPM model for further improvement.

2.2.3. One Class Cross Learning Model for Post-Processing and Fine-Tuning

However, the boundary is not yet accurate, and we present an effective one-class cross-learning framework to implement future fine-tuning. This framework is composed of two main sub-models: the Positive Only Learning Model (PL) and the Negative Only Learning Model (NL). Based on previous work, we can select high-quality negative samples for effective prediction and potential positive patches for positive learning [29]. In this way, we can obtain relatively effective negative patches.

In the Negative Only Learning Model, the background label is more prevalent than the lesion label, while in the Positive Learning Model, there are more lesion labels than background labels. Thus, the problem of imbalanced training samples between the two sub-networks can be addressed, which is one of the advantages of our model.

The output of the network (after SoftMax is applied) for each transformed image associated with the original one is a vector $\theta =({\theta }_{P+},{\theta }_{P-},{\theta }_{N+},{\theta }_{N-})$. The DeepLab classification model in this study, with the base model (ResNet-50), is trained to predict the patches, where $\theta$ represents the probability of the transformed image belonging to the new four classes, for example $j\in \{P^{+},P^{-},N^{+},N^{-}\}$.

We propose an inference process to fuse the outputs of these two transformed images $x_i^{+}$ and $x_i^{-}$ to predict the label of the original image $x_i$ based on the fusion of DeepLab twins. For each pair $(x_i^{+},x_i^{-})$, the prediction of the original image ${\widehat{y}}_i$ will be either P or N.

Thus, we obtain the trained cross-learning model, which is used on the test image set, employing the four sub-networks (Sub-Network A, Sub-Network B, Sub-Network C, and Sub-Network D) with the same structure via confidence, referred to as the positive reliability of the prediction. The learning model is enhanced by adding the confidence score, which mathematically corresponds to the probability of whether the data belongs to a positive class or not [30].

2.2.4. GPU Based Multiple Parallel Programs as the Base Layer

In hardware side, we explore a parallel model technique to train out models in an end-to-end fashion [31]. We load dataset from the cloud and make our parallel-based implementation of the proposed model.

Here, we refer to a GPU worker thread as a host thread that manages work dispatched to a single device for the compute-intensive function [32]. Some sequential code execution will be prior to parallel construct [33]. Parallelizable calculations in spatial data from the data pool are divided into individual tasks and performed using a pool of CPU and GPU worker threads that asynchronously consume the work on the host or the device [34], respectively. The sequential code forks N-1 threads specified by environment. We can parallelize 10 threads. Forking involves overheads of threads of thread creation and can slow down the code.

The master thread continues to process in parallel mode, and other starts processing their own chunk of data over multiple CPI’s [35]. In dynamic mode, threads are spawned slowly as each thread requests for chunks of data to process [36]. Since there is a possibility of a job being completed before spawning of all specified threads, it is necessary to choose the chunk size and number of threads wisely if the system is used for multiple parallel programs [37]. Iterations processed closer to the allocated memory location will complete faster, as compared to distant iteration processing nodes, due to memory access latencies [38].

3. Automatic Medical Image Segmentation for Thyroid Nodule Cancer Detection Based on Deep Dus-KFCM Clustering

In this section, Detection and classification of thyroid nodules are one of the challenging tasks in ultrasonography image analysis [39]. Lesion detection plays an important role in thyroid nodule diagnosis and follow-up [40]. Manual segmentation of TN images is a very time-consuming task and subject to intra- and inter-rater variability [41]. Accurate thyroid nodules lesion segmentation in abdominal sequences is an essential and crucial task for surgical planning and navigation in thyroid radiofrequency ablation [42]. However, thyroid nodule lesion segmentation in ultrasound image is a substantially challenging work because the intensity values of thyroid nodules are similar to those of adjacent structures [43].

In Figure 2, the idea is to combine two parts: Neural Network and Fuzzy Logic–Hybrid Neuro-Fuzzy system. The objective of this manuscript is to provide an efficient way to segment the thyroid nodules lesion area. This work focuses on design and development of an automated diagnostic system using thyroid nodules images. In this thesis, a coarse-to-fine method was applied to segment thyroid nodules lesion from ultrasonography images, which consists of two stages including rough segmentation and refined segmentation [44]. The rough segmentation is based on a kernel fuzzy C-means algorithm with spatial information (SKFCM) algorithm combined with spatial gray level co-occurrence matrix (GLCM) and the refined segmentation is implemented with deeplabv3+ (backbone with resnet50) algorithm to improve the overall accuracy.

Experimental results for thyroid nodules segmentation show that the method provides an accuracy of 87.9476% with specificity of 96.3343% and performs better than other related methods. The performance of the method was evaluated using two benchmark datasets including the thyroid nodules Segmentation and the non-cancerous datasets. Then use the real-time ultrasound strain elastography dataset to do the final test to verify robust sonographic features detection ability [45]. Our method achieves high accuracy in thyroid nodules lesion segmentation. The work describes an innovative way of using GLCM based textural features to extract underlying information in thyroid nodule cancer imagery. Modified deep DuS-KFCM endoscopy image segmentation method based on GLCM feature, The experimental results shown to be effective in image segmentation and has good performance of resisting noise, segmentation effect more ideal.

4. Research on Parallel and GPU-Accelerated Computing for Accurate Diagnoisis and Prognosis Prediction and Segmentation of Thyroid Nodule Cancer Recognition Using Patch Learning on Ultrasonography Image Automatic Medical Image Segmentation for Thyroid Nodule Cancer Detection Based on Deep Dus-KFCM Clustering

In this section, we propose a novel method for the thyroid incidentalomas detection with a small number of thyroid incidentalomas US images via fine-tuning LPLs model techniques. The proposed method newly annotates the regions of the TN and assigns three kinds of patch-level class labels considering not only the image-level GT (“ thyroidoma”/“non-thyroidoma”) but also the regions of the TN. Note that TN region annotation can be easily executed by non-clinicians. By using a LPLs model fine-tuned with patches to predict the patch-level class labels, we can select patches extracted from the inside of the thyroid gland for estimating the image-level class labels since the outside of the thyroid gland is unrelated to the image-level GT [46]. In the experiment, we use only 1,808 ultrasonography of thyroid nodules images for training and evaluate the detection performance with 615 ultrasonography of thyroid nodules images. Experimental results show that the proposed method achieves highly accurate detection with a small number of US-TN images for training.

As shown in Figure 2(B), the proposed pipeline encompasses four main components: Prepossessing and preparation of input data, feature extraction, fusion and feature abstraction, and classification segmentation.

This study proposes a DL model for detecting thyroid nodule lesions from fused GLCM and US data. GLCM map is a comprehensive analysis for describing lesion salience based on high-dimensional quantitative features extracted from the large quantity of medical images collected [47]. It has the potential to be an effective tool for personalized medicine based on feature descriptions of lesion from medical images [48].

The first component, that is, data preparation, includes the GLCM correction. The pixel map was filtered to create autoc, contr, corrm corrp and etc. The final map was created by interpolating each GLCM value map using the normalization method. The low-level features were calculated based on the image objects created via GLCM map and binary mark difference segmentations. The features extracted from the GLCM data and patch level endo images were fused at the low-level feature level.

In Figure 2B(a), we try to decomposition the TN-US image into patches: For each frame image in TN video clip, it needs to be divided into n non overlapping P×P patches, and these n patches are required to completely cover the whole slice, that is,

$$n={\displaystyle \frac{H\times W}{P^2}}$$

(1)

To identify potential lesion development, we assessed TN patches by incorporating GLCM data to capture influential aspects. We ensured patch dimensions aligned with US-TN resolution for thyroid nodule relevance. Each patch underwent analysis using various depth thresholds for comprehensive evaluation [40]. The program generated patch sequences based on specified conditions, forming a detailed list utilized in subsequent procedures [49].

In the next step, the matched positive and texture of all TN frame were formatted into patch vector files, with the patch selection established according to the predefined conditions. For each TN clip, a series of these patch vector files are generated, one file per patch. Each file contains the following for all patches: the patch ID, GLCM measurement of the TN patch under calculation. Due to the high dimensionality of the data, incorporating the individual pixel changes was not computationally feasible. Additionally, since our goal was to identify key regions likely to be TN hotspots, using the exact GLCM value at each pixel as opposed to a single GLCM value representation for the patch of the TN wasn’t expected to make a significant difference in the outcome of the study.

In Figure 2B(b), using the patch vector files as input, we performed statistical and grid search techniques to correlate the GLCM response as measured by the patch with the fixed size [50].

GT was applied to the patch in each captured frame as binary 0/1 vectors and we correlate the patch features composition to the image as a continuous sequence [51]. After jointing multi space attention(S), We classified embedding TN GLCM sensitivity against each patch as lesion or no-lesion corresponding to each image <or> GLCM search threshold, respectively [52]. GS avg search range was also utilized as a univariate analysis calculating odds-ratio and statistical significance to identify strong associations between presence or absence of a lesion [53].

These data mining models were applied to all sets of patch vectors, each set corresponding to a particular patch. Patches that included statistically significant patch sites by fussing GLCM map, normalizing fussing att., and achieved an overall accuracy of >5% by GS, were marked as significant and were considered for further evaluation. Notably, no correction for multiple accumulated searches was performed, as the goal was to be inclusive for discovering the most potential TN targets. Only patches identified as significant by the maximum statistics of the optimal evaluation index were used, with the hypothesis that patches with highest agreement by multiple indexes would be more likely to be substantially develop the TN lesion in Table 1, and therefore, more probable to be part of an FT network input.

In each significant patch, patch with significant correlation with the TN lesion identified were highlighted and stored as TN potential area that are impactful with regards to the TN prediction.

In Figure 2B(c), we examined a set of conditions for patch composition, as defined by the same patch size and resolution [54]. We then selected the condition that included patch threshold previously reported to be part of known TN and by eliminating conditions that resulted in redundant patch clusters. Patches were generated based on the selected range and evaluated by the evaluation indexes. We obtained patches determined to be significant by GS models, i.e., patches that contain statistically significant GLCM value range and high concordance of the elevation value with the predicted response by GS. To identify frequently occurring clusters of patches in these key patches which are likely to be important in TN binding as a network input, the frequency of occurrence of each patch across the significant multi evaluation value array was obtained.

5. Towards GS-Based One Class Cross-Learning for Accurate Segmentation of Thyroid Nodule Cancer Detection with Limited Source Labels

In this section, we describe our proposed method. We begin with a brief overview of the entire framework, followed by specific details. We introduced a deep learning solution for the problem of one-class classification, where only training samples of a single class are available during training. In the experiment, we use a real-world dataset consisting of 4 types of thyroid nodules images and 400 weakly supervised thyroid nodules images. Each thyroid nodules image contains a slide of a TN Lesion that is up to 108, 000 × 54, 000 × 3 pixels in size and was annotated by two thyroid nodule specialist. The supervised images have pixel-wise class labels, and the weakly supervised images only have a general ratio of cancer types. The supervised images are annotated into one non-cancer region and 4 cancer regions, PTC, Follicular Thyroid Carcinoma (FTC), medullary thyroid cancer (MTC) and Anaplastic thyroid carcinoma (ATC) [55]. In addition, there are vague areas in the lesion area, indicated in black, which were difficult for the acoustician and ultrasoundist to classify [56]. These difficult regions were excluded when evaluating the accuracy of the cancer type segmentation.

5.1. Description of The Process

5.1.1. Implementation Details

As shown in Figure 3(A), in negative learning, a randomly chosen class will most likely yield a correct label. Hence, its labels are less informative. In positive learning, a randomly chosen class will most likely yield an incorrect label. Hence, its labels are more informative. So negative learning labels are much more likely to be true than positive learning labels. Negative learning labels are much more likely to be true than positive learning labels, which tries to learn a classifier to model the distribution of positive data $X_P$ and negative data $X_N$ by using only positive labeled data and unlabeled data.

There exist a large number of two-step algorithms. The most primitive method is to consider all unlabeled examples as negative, including other types of thyroid nodules classes. A classifier trained with these negative examples and a positive data set may do some correct classifications since most of the unlabeled examples were already negative. On the other hand, since there are also positives in the unlabeled set, this classification may result in many false positives or false negatives. We implemented this method with the name ${\mathrm{SVM}}_{\mathrm{only}}$.

Seminal work led to six different kinds of choices: large positive-small positive (Pp), large positive-small negative (nP), large positive-large negative (PN). In prior work, the overall framework of the search strategy and study selection could refer to the section, and this is represented as the Positive Selectivity (PS) of the patch being assigned a pseudo label. PS is a parameter that determines how much of the images are given a positive pseudo label, and is defined by:

$$P{S}_c=(1-SAE)\times min(TC{R}_c,PC{R}_c)$$

(2)

Where $SAE$ is the distance between the ratios defined by Eq. 2, and $TC{R}_c$ and $PC{R}_c$ are the true cancer ratio and predicted cancer ratio, respectively. Eq. 2 is calculated for each class of thyroid ultrasound images, and positive pseudo labels are assigned to only $P{S}_c$ total patch images in the order of confidence.

As shown in Figure 3(B), this framework is composed of two main sub-models: the positive only learning model (PL for short) and the negative only learning model (NL for short) [57].

In Figure 4, firstly, we trim the image-set based on patching. In general, the class of interest is commonly called the positive class, and the rest are negative classes. In our study, we use the grid search model, GS for short, and with this model, we can select high-quality negative samples for effective prediction and potential positive patches for positive learning. This way, we can obtain relatively effective negative patches. In the negative only learning model, the background label outnumbers the lesion label. Conversely, in the positive learning model, there are more lesion labels than background labels [58]. Thus, the problem of imbalanced training samples in the two sub-networks can be addressed. This is one of the advantages of our model.

The output of the network (after SoftMax is applied) for each transformed image associated with the original one is a vector $\theta$. The DeepLab classification model in this study, using the base model (ResNet-50), is trained to predict the patches where $\theta$ represents the probability of the transformed image belonging to one of these four classes: $P_{+},P_{-},N_{+},N_{-}$. We propose an inference process to fuse the outputs of these two transformed images $x_i^{+}$ and $x_i^{-}$ to predict the label of the original image $x_i$ based on the Fusion of DeepLab Twins. For each pair $(x_i^{+},x_i^{-})$, the prediction of the original image $\widehat{y_i}$ will be either P or N.

Thus, we develop the trained cross-learning model, which is used in part B. In part B, these four sub-networks (Sub-Net A, Sub-Net B, Sub-Net C, and Sub-Net D) with the same structure are utilized in a one-class cross-learning model. Finally, the model provides an anomaly score via the confidence, which we refer to as the positive reliability of the prediction. This enables computers to learn effectively by incorporating the confidence score, mathematically corresponding to the probability of the data belonging to a positive class.

5.1.2. Construction of Prediction Models

The effectiveness of the proposed method is shown in results for AlexNet and VGG16-based backbone architectures. The performance of the proposed method is tested on publicly available datasets for abnormal image detection, novelty detection, and face-based mobile active authentication. The proposed method obtained state-of-the-art performance in each test case. The output of the network (after SoftMax is applied) for each transformed image associated with the original one is a vector

$$\theta =({\theta }_{P+},{\theta }_{P-},{\theta }_{N+},{\theta }_{N-}),$$

where ${\theta }_j$ is the probability of the transformed image belonging to class $j\in \{P+,P-,N+,N-\}$. Herein, we propose an inference process to fuse the outputs of the two transformed images $x_i^{+}$ and $x_i^{-}$ to predict the label of the original image $x_i$. In this way, for each pair $(x_i^{+},x_i^{-})$, the prediction of the original image $\widehat{y_i}$ will be either P or N. Let

$$\widehat{y_i^{+}}=argmax\theta =argmax({\theta }_{P+},{\theta }_{P-},{\theta }_{N+},{\theta }_{N-})$$

and

$$\widehat{y_i^{-}}=argmax\psi =argmax({\psi }_{P+},{\psi }_{P-},{\psi }_{N+},{\psi }_{N-})$$

be the ResNet-50 predictions for $x_i^{+}$ and $x_i^{-}$ respectively. Then: If $\widehat{y_i^{+}}=N+$ and $\widehat{y_i^{-}}=N-$, then $\widehat{y_i}=N$. If $\widehat{y_i^{+}}=P+$ and $\widehat{y_i^{-}}=P-$, then $\widehat{y_i}=P$.

If none of the above applies, then

$$\widehat{y_i}=\left\{\begin{array}{cc}N\hfill & \mathrm{if}max({\theta }_{Nj},{\psi }_{Nj})>max({\theta }_{Nj},{\psi }_{Nj}),j\in \{+,-\}\hfill \\ P\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(3)

Let computers learn well by adding the confidence score, which mathematically corresponds to the probability whether the data belongs to a positive class or not. They succeeded in developing a method that can let computers learn a classification boundary only from positive data and information on its confidence (positive reliability) against classification problems of machine learning that divide data positively and negatively.

6. Comparison with State-of-the-Art Methods

In this section, we provide detailed experiments and comparisons with state-of-the-art methods. We compare our method with segmentation models such as DeepLabV3+ (backbone: ResNet18), DeepLabV3+ (backbone: MobileNet), and DeepLabV3+ (backbone: Xception). The overall experimental results are depicted in Figure 5(a)–(l). We can clearly observe that:

False Positive/Negative Rates solve the problem where the training data only contain positive and reliable negative examples. The prediction results were quantitatively evaluated using three evaluations recommended for F-measure, False Positive Rate (FPR), False Negative Rate (FNR), Accuracy (ACC), and the Intersection over Union (IoU).

As shown in Figure 5, we find that the inherent problem of F-measure is improved; that is, the metric penalizes false negatives more heavily than false positives. This is mainly because our model can balance the sample size in positive learning and negative learning. We observe that in other existing methods, the sensitivity of the FPR is not high when the FNR changes vertically. Conversely, the bias effect of our model in this respect is lessened, and the overall F-measure remains relatively stable.

The F-measure remains high when FPR and FNR are at low values, reflecting the predictive robustness of our model. This result is also applicable to the evaluation indicators of F-measure, Accuracy, and IoU. Compared to other methods, our model shows a strong correlation between IoU and Accuracy, which reflects our model’s high sensitivity to IoU and robustness without significant changes. In other methods, the correlation between Accuracy and IoU does not exist; however, in our model, there is a certain correlation between IoU and Accuracy, which facilitates studying boundary accuracy.

7. Conclusion

The experimental results showed that the PM achieves a high patient-level gastritis detection performance with only a few annotations. This result also suggests the effectiveness of the automatic determination of threshold value to extract the lesion candidate region. The experiments show that the proposed method facilitates direct and overall analysis on thyroid ultrasound images and realizes highly accurate judgment of thyroid nodules location. This can help thyroid cancer diagnosis specialists more easily find missed lesion and decrease the thyroid carcinoma miss rate. It may be applicable in daily clinical practice to reduce the burden on physicians. The technology also allows high-risk precancerous lesions to be detected, which has the clinical and biological significance analysis on thyroid ultrasound finding. Experimental results on the constructed ultrasonography dataset demonstrate the significant improvement on the accuracy of our proposed framework compared with other state-of-the-art multi-instance learning methods.

In future research, our framework aims to predict genomic features of thyroid nodule through extensive patient data collection. The system has potential for detecting various diseases, including thyroid nodule-related conditions. Leveraging spatial features and temporal data enhances predictive capabilities for lesion-specific markers. Integrating diverse data sources, such as treatments, medical history, and diagnostic imaging, will improve model generalization and segmentation accuracy. And Humans struggle to integrate mixed data, extends the segmentation task by including multi-institutional data and introduces a new classification task. We could mix the different modality to improve our model generalization. For example, we could incorporate additional treatments into our model, including prescriptions, patient and family history, clinical notes, and reports. Furthermore, we could consider vital signs, lab diagnostics, genetics, second harmonic generation (SHG) microscopy, and the time-based clinical value of multi-gene testing.

Our research has several limitations. The experimental data used in this study were obtained from a single medical facility, which could affect the robustness of our method. Several studies have shown that extra data help increase the representation learning performance in self-supervised learning. Exploring the effect of data from multiple medical facilities on self-supervised learning will be one of our future works. Moreover, our previous study can improve the effectiveness and security of medical data sharing among different medical facilities, which fits well with self-supervised learning. And the major limitation of the proposed method is the small number of images. Training and evaluation of the proposed method were carried out using the data collected at a single facility only for comparison with our previous method. We plan to expand the dataset by including data from external facilities.