Segmentation of seventy-one anatomical structures necessary for the evaluation of guideline-conform clinical target volumes in head and neck cancers

1. Introduction

In the medical domain, the localization and determination of a disease’s extension can have a major advantage to the treatment. Ever since imaging modalities were available for cancer therapy, the precise delineation of organs and target volumes has been of great interest. The manual generation of these contours is thereby often time-consuming, requires intensive prior training and often lacks consistency between observers, especially for target volumes [1,2]. Because of the importance of available contour annotations in the clinical routine, a lot of research has been conducted in this area. Widespread early approaches that were used to automate medical image segmentation are atlas-based methods [3,4,5]. For this, reference images are first contoured to build the atlas. These atlas images are then registered onto the new image while the same deformation field is applied to the atlas’ contours, resulting in a segmentation of the new image. While this approach was proved to be successful in terms of manual labor reduction [6,7], it showed drawbacks in individual segmentation quality, when the image quality or the individual anatomy deviated from the atlas.

With the increase in deep learning (DL) methods that are capable of accurate contouring, the automatization of segmentation (auto-segmentation) is applied in more and more areas in which medical images are analyzed. The most popular network architecture for automatic medical image segmentation is the U-Net which was introduced by Ronneberger et al. [8]. The deployment of this architecture in a framework with self-configuring hyperparameters, the nnU-Net [9], increased the accuracy and accessibility of DL-based segmentation methods. With the nnU-Net, it is possible to train a state-of-the-art deep learning model for medical image segmentation tasks on custom data-label pairs, eliminating the need to explore task-specific hyperparameter settings.

While at first, DL methods were optimized to predict single volumes of interest, the importance of models for multi-organ segmentation increases [10,11]. Recently, the TotalSegmentator Version 2 toolkit was released1. The TotalSegmentator is a ready-trained open-access toolkit for the auto-segmentation of 117 anatomical structures in the whole body, which is based on the nnU-Net framework [11].

Multi-label segmentation models were shown to be beneficial for the segmentation accuracy of individual organs and robustness of the DL methods when compared to single label models [12]. Currently, most multi-organ segmentation models are trained on sparse labels (i.e. most voxels of an image are not labeled) due to missing dense annotations in available medical image data sets. Aiming to increase the segmentation accuracy, the dense segmentation of the human body is necessary, i.e. the segmentation of every anatomical structure and its substructures. Gare et al. [13] showed that for ultrasound images dense pixel-labeling improves disease classification when compared to models trained on only sparsely labeled images.

DL-based auto-segmentation enhances different tasks that need medical image segmentation. Enhancements can be in the form of improved standardization, time-savings or refined precision. Relevant tasks can be found in the realm of radiology, surgery [14], and radiotherapy. It also facilitates research fields like biomechanical modelling [15], and generation of synthetic medical image data sets [16], which in return improve the results in clinical applications. Nevertheless, the main application of automatic medical image segmentation methods lies within cancer diagnosis and treatment planning [17]. In cancer therapy, common auto-segmentation tasks are the segmentation of organs at risk (OARs) [18,19], target volumes [20,21,22,23], and metastases [24]. For example, Nikolov et al. [19] trained a DL-based auto-segmentation model that delineates 21 OARs achieving expert-level performance in the head and neck area.

In the field of radiation therapy, the exact contouring of OARs as well as target volumes is of major importance for the treatment outcome. Only with the precise delineation of target volumes and OARs, optimal tumor control can be achieved while adjacent healthy tissues are preserved. This significance is particularly pronounced in the head and neck region, where anatomical structures exhibit close spatial proximity paired with high anatomical flexibility. Target volumes as well as OARs are delineated by experts on the planning CT scans. These volumes are the basis for the objective function in the optimization of the radiation treatment plan.

Different target volumes are defined in radiotherapy. Following [25], the gross target volume is the visible and palpable, most inner tumor extension. It is surrounded by the clinical target volume (CTV) which comprises tissue that is potentially infiltrated by microscopic tumor cells. The CTV can itself be subdivided into the primary CTV and the nodal CTV. The primary CTV is drawn as a margin of 0.5 – 1 cm around the gross target volume, while the nodal CTV follows the lymphatic pathways and includes all areas that are found to harbor microscopic tumor cells with a probability of 10% or more [26,27,28]. The outermost target volume is the planning target volume which surrounds the union of all former mentioned target volumes and compensates for beam parameter uncertainties, patient placement errors, organ fluctuations and other motion-induced variance [29].

The extension of the CTV is not visible with modern imaging techniques, since it comprises normal tissues infiltrated by microscopic tumor cells. The definition of its outline is rather based on recurrence studies and thus, empirically built clinical experience [30,31]. This makes the delineation of CTVs a difficult task for clinicians that needs many years of training [32]. Its complexity is not only visible in the training needed to perform this task, but also in the time needed to produce acceptable delineations and in their resulting divergence. Given the same CT scan, the manual CTV delineations of different experts show a large inter- and intra-observer variability of up to 200% difference in volume [1].

The quality of manual labels heavily affects the training and thus, the prediction accuracy of supervised learning methods. The inconsistent manual delineations of CTVs have a negative impact on the auto-segmentation of target volumes [33,34]. For that, researchers in this field focus on curating consistent data sets by executing extensive peer-reviews on the process of manual contouring or incorporating contours of only a minimum number of clinical experts, or institutes [21,22,23]. For CTV delineation, the predicted labels are reported to still need intensive pre- and post-processing [35,36,37,38] and they are not easily adaptable to changes in segmentation standards or patient-individual requirements. All this is done, aiming for improved spatial conformance of the predicted contour with manual delineation, while knowing that manual delineations are not well standardized.

Not only the comparison to labels that are highly dependent on the expert that generated the label, but also recent studies on evaluation metrics raise critiques on the current state-of-the-art. Reinke et al. [39] point out that the measurements of pure spatial overlap (i.e. the DICE) do not necessarily quantify the actual quality of interest in medical image segmentation tasks. For the delineation of CTVs the quality of interest that should be measured is the conformance of the CTV delineation with the expert guidelines.

To overcome the variety in CTV delineation, the detailed clinical knowledge about the extension of the CTVs is collected in international consensus expert guidelines including head and neck treatments [27,28]. These expert guidelines provide a commonly accepted delineation scheme for the CTVs in a rule-based manner and thus, standardize their segmentation. As one example, Grégoire et al. [27] focus on the delineation of nodal CTV in the head and neck area. In these expert guidelines, the nodal CTV is subdivided into ten levels with some additional subdivisions. The extent of each single level is described by bordering anatomical structures. Thus, the expert guidelines convert the difficult problem of delineating the extent of cancerous infiltration which is not visible in CT scans, in a contouring task of anatomical structures. The selection of levels that should be irradiated is based on the location of the primary tumor.

Oriented towards the goal of evaluating guideline conformance of CTV delineations, in this study, the 71 most important anatomical structures mentioned in the expert guidelines have been chosen for an auto-segmentation task. For that, all 71 structures have been manually delineated, and used to train nnU-Net models for auto-segmentation. The predictions for 18 unseen data sets are evaluated against the manual labels as well as segmentations generated by the TotalSegmentator, and compared to previously reported segmentation results. So far, studies on the segmentation of anatomical structures have only published results on a small subset of the necessary 71 anatomical structures. The existent results are widely distributed over multiple unrelated publications. Finally, the impact of the segmentation accuracy on the construction of CTV delineation according to the expert guidelines is discussed.

2. Materials and Methods

2.1. Image properties of the data set

The planning CT scans for this study were aggregated from four different study cohorts. Figure 1 shows an exemplary CT scan of each cohort. All patients received radiotherapy for head and neck cancer. For each patient, there was exactly one planning CT scan considered in this study. Each CT scan consists of 90 to 220 single slices (mean: 141 ± 24) of 512 × 512 voxels each. The voxel size ranged from 0.98 × 0.98 × 2 ${\mathrm{mm}}^3$ to 1.27 × 1.27 × 3 ${\mathrm{mm}}^3$.

The training data set and test data set are mutually exclusive. The training data set (86 scans) included (a) 84 in-house HNC patients from three different cohorts (varying setup, positioning, devices, and protocols) [43,44], and (b) 2 open access HNC data sets [40,41,42]. The test data set (18 scans) is curated from the same three study cohorts (14, and 4 scans, respectively). The patient selection for the test data set was based on available meta-information to best represent the variety of the data cohorts. Factors for the selections were study cohort, location of the primary tumor, gender, presence of a tracheostoma, size of nCTV, estimated age and weight of the patient.

2.2. Label selection and generation of the manual labels

The 71 structures were chosen based on their number of occurrence in the Grégoire et al. [27] expert guidelines. The resulting set of anatomical structures is visualized in Figure 2. Manual labels of the 71 anatomical structures were generated for all 104 CT scans by six different trained observers on a Wacom Cintiq 24HD Display in RayStation 8B(R) SP1. The observers were following a standard operation procedure for the delineations that included (a) the unambiguous definition of the structures’ extent (e.g. mandible without teeth), (b) windowing, and (c) spatial restrictions based on other anatomical structures (mostly cranial and caudal). The whole standard operation procedure can be found in Appendix A.1 . Each data set was at least once reviewed and if necessary adjusted by one of the other observers before it was accepted for the study.

For one patient data set, 41 selected structures were segmented a second time by one of the trained observers who was not involved in the initial segmentation or the review of this patient. Based on those two sets of contours, the inter-observer variability was approximately assessed.

Caused by the field of view of our CT scans, the esophagus, the sternum (corpus and manubrium), the lobes of the lung, the trachea, the trapezius muscles, the brachiocephalic veins, and the skin are never or not always completely present on our patient scans, but cut off on the caudal edge of the scan. The sternum corpus is sometimes not present at all. Further, in cases where the patients were post-operatively irradiated, or the extension of the primary tumor distorted surrounding anatomical structures, the respective missing anatomical structures were not segmented. In total, there were 30 anatomical structures missing. Fifteen of those structures cumulated in two test patients (#8, #7), and three other patients had at least two missing structures. Nine of the 18 test patients were not missing any structure and thus, had the full set of 71 anatomical structures manually segmented.

2.3. Network training and label prediction

For the automatic segmentation, the nnU-Net framework Version 1 was chosen and trained with one adaption to the default parameters: mirroring was removed from the data augmentation to keep the left-right orientation of the patients consistent during training. The final training data set provided for the nnU-Net training was generated by mirroring all 86 training data sets. Left and right instances of anatomical structures were then swapped back for left-right consistency after mirroring.

Since in the nnU-Net Version 1, a network can only be trained for non-overlapping structures, the labels of all 71 anatomical structures were subdivided into three non-overlapping, disjoint subsets, containing (a) the labels for all bones, muscles, vessels, air-related structures, glands and the esophagus (#64), (b) the labels for all cavities (#6), and (c) the skin label (#1). According to the author, nnU-Net Version 2 has no accuracy advantages over its Version 1 [45].

Following the nnU-Net’s five-fold cross-validation standard, for all three subsets there were five 3D full-resolution models trained with the trainer V2. Fold 1 and fold 2 were using 137 data sets for training and 35 data sets for validation, while fold 3 – 5 were using 138 data sets for training and 34 data sets for validation. Each fold was trained for 1000 epochs. The predictions were made for all 18 previously unseen test data sets in the nnU-Net’s default 5-heads manner. No postprocessing was applied.

All computations were executed using the nnU-Net Version 1.7.0 with Python Version 3.9.7, PyTorch 1.10.2 with CUDA Version 11.3.1. Training and predictions were executed on a computer with an AMD Ryzen™ 9 3900X Processor, 128 GB RAM, with an NVIDIA GeForce RTX 3090, and 24 GB VRAM.

For 16 of our anatomical structures, segmentations can also be retrieved by using the pre-trained TotalSegmentator toolkit. We employed the TotalSegmentator as Python library on our 18 test patients with default configurations. The predictions generated by the TotalSegmentator were run on a computer with an Intel® Core™ i7 Processor, 64 GB RAM, with an NVIDIA GeForce RTX 2070, and 8 GB VRAM.

2.4. Evaluation of predicted labels

We assess the similarity and distance between two distinct labels of the same structure through three metrics: (a) their volumetric overlap, measured using the Sørensen–Dice coefficient (DICE) [46,47], (b) the distance between both contours, evaluated by the Hausdorff distance (HD) [48] and (c) the fraction of deviation larger than 2 mm, quantified using the surface DICE (sDICE) as defined in Nikolov et al. [19]. For the evaluation of the HD we chose the 95th percentile (HD (95)). The margin of 2 mm is chosen for the sDICE because of its clinical relevance in radiation therapy with photons. The sDICE (2 mm) is considered to indicate the correction effort needed for the predicted CTVs. This selection of metrics is consistent with the metrics reloaded framework [39]2. Structures that are not present in the manual labels, in the predicted labels or both sets of labels are left out in the analyses. For the calculation of all metrics, the library surface-distance-based-measures Version 0.1 was used.

3. Results

3.1. Analysis based on Volumetric Overlap

An overview of the volumetric overlap between the manually segmented and the predicted anatomical structures is given in Figure 3. It shows the mean DICE (DICE_m) value for each anatomical structure over all test patients grouped by their tissue types. The median and standard deviation of the DICE_m is 0.88 ± 0.09 for air-related structures, 0.84 ± 0.07 for bones, 0.77 ± 0.08 for cartilages, 0.78 ± 0.02 for glands, 0.78 ± 0.09 for vessels, and 0.63 ± 0.16 for muscles. Outliers are left and right internal carotid arteries. The box plot of all muscles is wide spread, while all other box plots show a centered median with symmetric and narrow distribution of DICE_m values around it. The analysis will focus on structures that are below the 25th percentile (Q1) in DICE_m within the group of muscles. This comprises all single parts of the constrictor muscle, the right digastric muscle, the left and right posterior scalene muscles, and the left thyrohyoid muscle.

A precise evaluation of the volumetric overlap between the manually segmented and the predicted anatomical structures is given in Table 1. It shows the DICE_m value for each anatomical structure over all test patients, as well as the inter-observer variability in DICE and previously reported DICE values for comparison. Some of the individually segmented 71 anatomical structures form a meaningful unit together, i.e. they are substructures of a coherent anatomical structure. Thus, Table 1 also contains (a) the sternum (M., C.), a combination of the sternum manubrium and the sternum corpus, (b) the constrictor muscles (s., m., i.), a combination of the inferior, the middle and the superior constrictor muscle, (c) the right and left scalene muscles (an., me., p.), a combination of the right and left anterior, medius and posterior scalene muscle, respectively, and (d) the pharynx (nasop., orop., hyp.), a combination of the nasopharynx, hypopharynx and oropharynx. With these combinations, Table 1 contains a total of 76 anatomical structures.

The inter-observer variability is approximated for 45 selected structures and their available combinations. Inter-observer values outside the 3$\sigma$ interval around the DICE_m are indicated by an asterisk (*). Although within the 3$\sigma$ interval, the inter-observer DICE is noticeably low for the left internal carotid artery, the left and right posterior scalene muscles, the left and right digastric muscles, and the tonsils.

Table 1 also shows previously reported DICE_m values. While for most structures, there is no DICE value found for comparison (48 of 76 structures), or only a single reference (17 of 76 structures), there are multiple comparisons for 11 anatomical structures. Detailed values for multiple comparisons are listed in Appendix A.2. Our prediction results are mostly within the 3$\sigma$ interval (single comparison) or within the given range (multiple comparisons). Lower DICE_m values than previously reported result from the internal carotid arteries, and the inferior, middle and superior constrictor muscle. For the former, left and right instances are jointly evaluated in Nikan et al. [49], Ke et al. [50], while for the latter, our results are comparable to Thomson et al. [51], Van Dijk et al. [52] when all substructures are combined. Higher DICE_m values than previously reported result from the levator scapulae muscles, and the prevertebral muscles, and the sternum (M. C.), which is not completely present on our CT scans.

3.2. Analysis based on Distance-Based Metrics

An overview of the distance-based metrics between the manually segmented and the predicted anatomical structures is given in Figure 4. It shows the mean HD (95) (HD_m) and the mean sDICE (2 mm) (sDICE_m) for each anatomical structure grouped by their tissue type. The median and standard deviation of the HD_m is 4.96 ± 2.22 for air-related structures, 3.15 ± 1.51 for bones, 4.28 ± 1.88 for cartilages, 5.04 ± 0.67 for glands, 7.53 ± 4.13 for vessels, and 7.29 ± 4.23 for muscles. The median and standard deviation of the sDICE_m is 0.90 ± 0.04 for air-related structures, 0.94 ± 0.03 for bones, 0.89 ± 0.07 for cartilages, 0.85 ± 0.04 for glands, 0.87 ± 0.05 for vessels, and 0.86 ± 0.13 for muscles. Outliers in HD_m are the right platysma muscle and the right posterior scalene muscle. The outlier in sDICE_m is the tongue.

For the HD_m, the analysis will focus on structures that are above the 75th percentile (Q3) within the group of vessels and the group of muscles. This comprises the right internal carotid artery, the left and the right subclavian artery, the right sternocleidomastoid muscle, the superior constrictor muscle, the left platysma muscle, and the left posterior scalene muscle. For the sDICE_m, the analysis will focus on structures that are below the 25th percentile (Q1) within the group of vessels and the group of muscles. This comprises the left and the right internal carotid artery, the right subclavian artery, the middle and the superior constrictor muscle, the left and the right digastric muscle, and the left and the right posterior scalene muscle.

A precise evaluation of the distance-based metrics between the manually segmented and the predicted anatomical structures is given in Table 2. It shows the HD_m and the sDICE_m for all 71 segmented anatomical structures and the five combinations over all test patients, as well as the inter-observer variability in HD (95) and sDICE (2 mm). The inter-observer variability is calculated for the same subset as described for the DICE. Inter-observer values outside the 3$\sigma$ interval around the HD_m and sDICE_m, respectively, are indicated by an asterisk (*). Although within the 3$\sigma$ interval, the inter-observer HD (95) is noticeably low for a variety of scalene muscles, and the tonsils. For the DICE and sDICE (2 mm), structures of low overlap are the same.

3.3. Completeness of predicted label set

In the 18 test patients’ anatomies, a total of 30 anatomical structures are absent. Thirteen of these 30 structures were correctly identified as missing anatomical structures by the trained nnU-Net models (true negatives). The remaining 17 missing structures were erroneously contoured (false positives). Amongst these 17 structures the sternothyroid muscle was contoured five times, the platysma muscle three times, and the posterior scalene muscle two times.

The analysis of anatomical structures that were present in the test patients’ anatomy, but not segmented by the trained nnU-Net models (false negatives), result in the model’s capability to predict all but two of the present structures (larynx (air), posterior scalene muscle (l)). The tonsils were excluded from this analysis, since they are generally difficult to segment as indicated by the inter-observer variability which is shown in Table 1 (DICE) and Table 2 (HD, sDICE). They were predicted correctly on both sides only in eleven of the 18 test patients. Even when predicted, the overlap between manual and predicted segmentations was small.

3.4. Analyzing only patients without tracheostoma

In the training data set, approximately one third of the patients were scanned with a tracheostoma. In the test data set this ratio is one sixth, respectively. Although trained on several data sets with tracheostomy, test patients that have a tracheostoma show below-average values in several anatomical structures. Table 3 lists the 17 most deviating structures. For these structures, the DICE_m, HD_m and sDICE_m is shown when only patients without tracheostomy are considered. The deviation of all metrics between this analysis and the analysis considering all patients is presented in brackets. All structures beside these 17 anatomical structures show low deviations between both analyses: the average deviation is 0.00 ± 0.07 in DICE_m, and -0.01 ± 0.07 in sDICE_m.

Table 3. Mean DICE, mean HD (95) and mean sDICE (2 mm) for all test patients without tracheostomy (#15). Seventeen structures are selected for that the mean DICE and mean sDICE (2 mm) increased the most when compared to the values resulting from the analysis including all patients. The deviation between the analysis including all patients and the analysis excluding patients with tracheostomy is given in brackets.

Structure	DICE	HD (95)	sDICE (2 mm)
Trachea	0.92 (0.13)	5.64 (-7.40)	0.93 (0.16)
Hyoid Bone	0.83 (0.12)	2.31 (-7.32)	0.96 (0.09)
Thyroid Gland	0.84 (0.14)	5.90 (-1.32)	0.92 (0.18)
Internal Carotid Artery (r)	0.57 (0.10)	11.77 (-12.50)	0.77 (0.10)
Internal Jugular Vein (r)	0.78 (0.15)	8.09 (-0.98)	0.89 (0.13)
Constrictors (s., m., i.)	0.59 (0.19)	7.14 (-0.32)	0.90 (0.10)
Middle Constrictor	0.48 (0.21)	9.17 (-2.93)	0.75 (0.15)
Superior Constrictor	0.52 (0.23)	11.32 (0.50)	0.75 (0.14)
Digastric (r)	0.51 (0.30)	7.56 (-5.75)	0.69 (0.33)
Platysma (r)	0.54 (0.18)	17.61 (-15.24)	0.78 (0.20)
Sternothyroid (r)	0.60 (0.21)	4.66 (-3.01)	0.91 (0.28)
Sternocleidomastoid (l)	0.86 (0.12)	3.63 (-7.86)	0.93 (0.09)
Sternocleidomastoid (r)	0.85 (0.26)	5.17 (-42.80)	0.92 (0.26)
Thyrohyoid (r)	0.57 (0.09)	2.85 (-1.79)	0.91 (0.12)
Esophagus	0.82 (0.12)	5.41 (-4.44)	0.90 (0.11)
Hypopharynx	0.68 (0.23)	5.95 (-4.73)	0.86 (0.18)
Soft Palate	0.63 (0.16)	8.64 (-4.12)	0.78 (0.14)

Table 3. Mean DICE, mean HD (95) and mean sDICE (2 mm) for all test patients without tracheostomy (#15). Seventeen structures are selected for that the mean DICE and mean sDICE (2 mm) increased the most when compared to the values resulting from the analysis including all patients. The deviation between the analysis including all patients and the analysis excluding patients with tracheostomy is given in brackets.

Structure	DICE	HD (95)	sDICE (2 mm)
Trachea	0.92 (0.13)	5.64 (-7.40)	0.93 (0.16)
Hyoid Bone	0.83 (0.12)	2.31 (-7.32)	0.96 (0.09)
Thyroid Gland	0.84 (0.14)	5.90 (-1.32)	0.92 (0.18)
Internal Carotid Artery (r)	0.57 (0.10)	11.77 (-12.50)	0.77 (0.10)
Internal Jugular Vein (r)	0.78 (0.15)	8.09 (-0.98)	0.89 (0.13)
Constrictors (s., m., i.)	0.59 (0.19)	7.14 (-0.32)	0.90 (0.10)
Middle Constrictor	0.48 (0.21)	9.17 (-2.93)	0.75 (0.15)
Superior Constrictor	0.52 (0.23)	11.32 (0.50)	0.75 (0.14)
Digastric (r)	0.51 (0.30)	7.56 (-5.75)	0.69 (0.33)
Platysma (r)	0.54 (0.18)	17.61 (-15.24)	0.78 (0.20)
Sternothyroid (r)	0.60 (0.21)	4.66 (-3.01)	0.91 (0.28)
Sternocleidomastoid (l)	0.86 (0.12)	3.63 (-7.86)	0.93 (0.09)
Sternocleidomastoid (r)	0.85 (0.26)	5.17 (-42.80)	0.92 (0.26)
Thyrohyoid (r)	0.57 (0.09)	2.85 (-1.79)	0.91 (0.12)
Esophagus	0.82 (0.12)	5.41 (-4.44)	0.90 (0.11)
Hypopharynx	0.68 (0.23)	5.95 (-4.73)	0.86 (0.18)
Soft Palate	0.63 (0.16)	8.64 (-4.12)	0.78 (0.14)

3.5. Comparison to TotalSegmentator

Applying the pre-trained TotalSegmentator framework (TS) to our data resulted in predictions of 16 common anatomical structures. Thereby, our label ‘Brachiocephalic Artery‘ corresponds to their ‘Brachiocephalic Trunk’. All 16 structures are listed in Table 4 which shows the DICE_m comparing the TS predictions with our manual segmentations. Differences between this comparison and the comparison of our predictions to the manual labels are favoring segmentations generated by our models (i.e. all values are negative). Below the Q1 of -0.10 for the difference in DICE_m is the trachea, the thyroid gland, and the left and right common carotid arteries.

Table 5 shows the same comparisons using the HD_m and the sDICE_m. All predicted segmentations generated by our models show better results in HD_m (i.e. all diff. values are positive) and better or equal results in sDICE_m (i.e. all diff. values are negative or zero). Above the Q3 of 7.98 for the difference in HD_m is the trachea, the left and right common carotid arteries, and the right subclavian artery. Below the Q1 value of -0.09 for the difference in sDICE_m is the trachea, the thyroid gland, and the left and right common carotid arteries.

4. Discussion

When comparing the grouped DICE_m between tissue types, groups with good contrast on CT scans like air-related structures and bones show an increased accuracy when compared to other groups. Noticeably, the variation in DICE_m is the largest for the group of muscles. First, this group has the largest number of different anatomical instances. Further, the contrast of soft tissues on CT scans is not sufficient to identify most muscles completely. Finally, the group of muscles is also the most diverse group ranging from structures with an average volume of 550 voxels (digastric muscle) to 55,000 voxels (trapezius muscle).

4.1. Reasons for impaired prediction accuracy

We have visually analyzed cases of impaired prediction accuracy for highlighted anatomical structures from before. In the following section, reasons for this impaired prediction accuracy are summarized.

The visual analysis of cases in which the internal carotid artery (ICA) shows especially low DICE and sDICE on both sides, results in four common reasons for deviations between the manual segmentation and its prediction: (a) the ICA is a thin structure, (b) the transition between internal carotid artery and common carotid artery varies, (c) the final slice, on which the ICA occurs cranially varies, and (d) due to metal in the mouth, CT artifacts occur in this area. Figure 5 shows the deviation between manual and predicted segmentation of the ICA due to inconsistent decision on the most cranial slice and the bottom row of Figure 6 shows metal artifacts.

For the subclavian artery similar reasons are resulting in small DICE_m and sDICE_m: (a) the subclavian artery is a thin structure, (b) the transition between the right subclavian artery and the brachiocephalic artery varies, and (c) the lateral extension varies.

The visual analysis of the superior constrictor muscles and middle constrictor muscles also results in clear confusion at the area of transition between both structures, as well as the transition between the middle and the inferior constrictor muscles. This observation is supported by the above-median performance of their combination (i.e. constrictors (s., m., i.)). Training their combination, and differentiating the substructures in a rule-based post-processing, might be beneficial to the auto-segmentation of the constrictor muscles and similar cases.

The digastric muscles and the posterior scalene muscles show an (almost) below Q1 performance in DICE_m and sDICE_m with large standard deviations amongst test patients. DICE values range from [0 – 0.83] for the digastric muscles and [0 – 0.71 (0.81)] for the posterior scalene muscles. sDICE values deviate by more than 0.68 (digastric muscles) and 0.85 (posterior scalene muscles) between minimum and maximum. All predictions show greater accordance with the manual labels than the segmentations generated by the second observer (high inter-observer variability).

The tongue has an above-median DICE_m, but a noticeable low sDICE_m. Since the tongue is a theoretically easy to locate structure of above-average volume, the DICE_m does only marginally indicate problems with its segmentation. The sDICE_m signals inconsistencies in the precise outline of the tongue. Reasons are metal artifacts that occur predominantly in the area of the mouth which impair the precise segmentation of the tongue.

The right platysma muscle is an outlier in HD_m. The analysis of individual cases shows a deviation of the manual labels in the frontal-dorsal direction and the cranial-caudal direction. Since the platysma muscle is a thin cutaneous muscle, it is sometimes barely visible in its most frontal and most dorsal extension. Thus, the network is trained on only a few extended examples. Auto-segmentations depict only the mostly visible inner extension of the platysma muscles.

4.2. Inter-observer variability, and tracheostomy analysis

The anatomical structures with an inter-observer variability outside the 3$\sigma$ interval around the mean in any of the three metrics or a value below the Q1 in DICE_m or sDICE_m or above the Q3 in HD_m were visually analyzed. Two systematic reasons are found that explain deviations. First, the lateral extension of the subclavian artery was inconsistent. Second, muscular structures were systematically segmented wider by one observer than by the other. This holds for the prevertebral muscles, the sternocleidomastoid muscles, the trapezius muscles and the digastric muscles. The deviation between all scalene muscles and the tonsils did not follow systematic reasons. Those structures are barely or not visible in the planning CT scans. Figure Figure 6 shows this for the tonsil (green arrows). This results in largely deviating contours between both observers as visualized in the right column of Figure 7. No unambiguous reason can be given for the right internal carotid artery. As it is a thin structure that is difficult to segment, deviations occur in some central slices, while its left counterpart is much better aligned between both observers. No clear difference is visible between both sides of the patient CT scan.

Although the DL-models were trained on a distinct amount of patient data sets with tracheostomy, leaving out those patients from the analysis improves seventeen selected structures noticeably in almost all of the three metrics. Analyzing the deviation of the DICE_m and the sDICE_m for all other anatomical structures shows almost no change. Most of the 17 structures are in close proximity to the tracheostomy or the distortions in the larynx caused by tracheostomy.

4.3. Comparison to TotalSegmentator

Most anatomical structures that are automatically segmented by the TotalSegmentator framework (TS) are very similar to our own generated segmentations. For those structures that are deviating noticeable there is a common reason when analyzing the segmentations visually. Figure 5 includes the 3D comparison of those structures. The most common reason is the disagreement in the starting and ending position of elongated structures like the common carotid artery, the trachea, and the subclavian artery. Our manual segmentations for the common carotid arteries ends cranially at the artery’s bifurcation. Although caudally starting very similarly, the segmentations of the TS end approximately half way to the artery’s bifurcation, close to the cranial edge of the esophagus and the trachea. For the trachea, our manual labels exclude the bronchi, while the TS predicted segmentations include the right and left primary bronchi. Our manual labels for the subclavian artery exceed the TS generated labels laterally.

Deviations in the auto-segmentation of the thyroid gland result from patient-individual differences, rather than a systematic difference in the definition. Especially in patients that are equipped with a tracheostoma, the TS predictions deviate more from the manual segmentations than our own predictions. It might be, that in the training data set on which the TS model was trained, there were less or no patient data with a tracheostoma.

4.4. Impact on CTV delineation

The delineation of CTVs should be targeted for auto-segmentation using DL algorithms. Following the international consensus guidelines of Grégoire et al. [27]. this study can be the basis for improved standardization and reduced workload. In the following section, the implications are analyzed that the prior described systematic deviations in the auto-segmentations of anatomical structures have on the clinical target volume delineation when following Grégoire et al. [27].

The predicted contour of the internal carotid artery (ICA) deviates caudally when transitioning into the common carotid artery (CCA) and its final slice cranially, as well as due to metal artifacts. Within the expert guidelines [27], the ICA is needed as the medial edge of Level II, the lateral edge of Level VIIa, and the medial edge of the Level VIIb. All these levels are transitioning into each other and the precise boundary becomes only relevant if some, but not all of these levels are irradiated. Since Level II begins caudally approximately where the CCA and ICA are transitioning, one might add the CCA as boundary into the rules when automating the delineation of Level II. The cranial edge of Level II is given by either the lateral process of C1 which the ICA always exceeds, or Level VIIb. The cranial edge of Level VIIb is the base of skull (jugular foramen) which the ICA reaches in all our test patients. Thus, the deviations introduced by the auto-segmentation of the ICA do not affect the CTVs’ delineation.

The predicted contour of the subclavian artery (SuA) deviates laterally and in its transition to the brachiocephalic artery. Within the expert guidelines [27], the SuA is needed as the posterior edge of the Level IVb. Caudally, this posterior boundary is cumulating both, the SuA and the brachiocephalic artery, such that their transition does not affect the delineation of the CTV. Also cranially, the lateral deviation of the SuA’s segmentation does not affect the posterior edge of the Level IVb. This is, because the SuA’s extension always exceeds the necessary boundary of Level IVb.

The predicted contour of the inferior, middle and superior constrictor muscles (CM) deviates caudally and cranially at the transitions between each other. Within the expert guidelines [27], the CM is needed as the anterior edge of Level VIIa which is bordering the superior or middle pharyngeal constrictor muscle. This boundary is cumulating both, the superior and middle CM, such that their transition does not affect the delineation of the CTV.

The predicted contour of the platysma muscle (PM) deviates in frontal and dorsal direction as well as in cranial and caudal direction. Within the expert guidelines [27], the PM is needed as caudal edge of Level Ia and Ib, lateral edge of Level Ib and Level V, and anterior edge of Level VIa. The caudal edge of Level Ia required sufficient delineations of the PM in its central regions which is shown consistently. The caudal edge of Level Ib is described by a plane independent of the PM. The PM only cuts this plane as it is the lateral border of Level Ib. For this, the central parts of the PM are relevant. Those are well-predicted. In the boundary descriptions of Level V and Level VIa, the skin is given as an alternative edge. Since the PM is a thin cutaneous muscle, the expert guidelines already account for its potential invisibility. In this case, there will be no further implications for the CTV delineation than the irradiation of the PM itself.

The predicted contour of the anterior belly of the digastric muscle (aDM) deviates unsystematically. Within the expert guidelines [27], the aDM is needed as caudal and lateral edge of Level Ia, and medial edge of Level Ib. For the caudal edge of Level Ia the aDM is not the primary boundary, but a substitute for the PM if the PM is not visible. Due to inconsistent delineations of the sDM, substituting the PM in this case might cause deviations in the caudal boundary of Level Ia. Nevertheless, as discussed before, the PM is often delineated well in the discussed region. Visually analyzing the data, as lateral edge of Level Ia, often the mandible is chosen. Further, as medial edge of Level Ib, often the Level Ia is chosen. Thus, the delineations we got from the clinics do not always spare the aDM. With our inconsistent delineations, we cannot improve this situation and spare the aDM reliably. No solution can be provided for cases in which Level Ib is irradiated while Level Ia is not.

The predicted contour of the posterior scalene muscle (pSM) deviates unsystematically. Within the expert guidelines [27], the scalene muscles are needed as medial edge of Level II , Level III, Level IVa, Level V, Level Vc, posterior edge of Level IVa, and lateral edge of Level IVb. Although not specified precisely, the visual analysis shows that most boundaries are given by the anterior scalene muscle. The pSM potentially plays a role in delineating the medial edge of Level V caudally. Here, the confusion between different scalene muscles does not affect CTV delineation, but the pSM could be unintentionally irradiated if contoured erroneously.

The predicted contour of the tongue and the tonsils deviate unsystematically due to metal artefacts and missing soft tissue contrast. Since both structures are not used as a boundary definition, but only as selection criterion for nodal levels in the expert guidelines [27], the CTV delineation is not affected by distortions of these two structures.

4.5. Limitations

In our study, we segmented 71 anatomical structures. With additional tools like the TotalSegmentator, the set of structures can be further extended. Nevertheless, even including multiple models, there are still anatomical structures that are segmented neither previously nor in this study. Thus, the dense segmentation of all anatomical structures in the human body is still at issue. Even if dense annotations become more feasible, the large inter-observer variability indicates upcoming problems related to this topic. Before dense annotations can be generated, better agreement of structures’ definitions should be reached. Their precise delineation could be supported by additional multi-model images.

Not all necessary structures are covered for the auto-segmentation of all CTV levels in the head and neck area. Structures like the posterior belly of the digastric muscle, the mylo-hyoid muscle, the transversal cervical vessel and the infrahyoid (strap) muscles are missing for completeness. Further, some segmented structures do not lead to sufficient prediction accuracy to be spared (e.g. the anterior belly of the digastric muscles).

Although our training data set was very diverse, the number of training and test samples was too low to train the models to identify each image feature and each patient condition. Thus, patients with tracheostomy led to worse segmentation accuracies. The same might hold for postoperative patients, different stages of contrast agents, or different resolutions of CT scans.

5. Conclusions

In this study, we have automatically segmented 71 anatomical structures in the head and neck area relevant for CTV delineation. Most of these structures have not been automatically segmented before. We analyzed systematic deviations of anatomical structures that showed mean DICE, mean HD or mean sDICE values below the Q1 (above the Q3, respectively) and their impact on the automation of CTV delineation. No deviation is expected to be inferior to the current clinical practice.

These results are a step forward towards dense annotations and the auto-segmentation of guideline conform CTV delineation.

Appendix A.1. Standard Operation Procedure

• Nasopharynx
  • Cranial boundary: up to the nasal septum
  • Caudal boundary: from the hard palate
• Oropharynx
  • Cranial boundary: from the hard palate
  • Caudal boundary: epiglottis
• Hypopharynx
  • Cranial boundary: epiglottis
  • Caudal boundary: transition to the esophagus, along with the caudal end of the cricoid cartilage
  • Segmentation note: No clear caudal boundary, orientation is based on the larynx-air structure
• Tongue (muscle)
  • Bounded by the oral cavity
  • Caudal boundary: tongue base (ambiguous border)
• Thyroid cartilage
  • Segment in the larynx window
  • Boundary: entire cartilage structure (typical shape was always recognizable in 3D view)
• Sternocleidomastoid muscle
  • Cranial boundary: mastoid cells, up to the skull
  • Caudal boundary: clavicle and sternal manubrium, occasional branching near the origin may be visible
• Thyroid gland
  • Bright structure, merging caudally, variable cranial boundary
• Hyoid bone
  • Segment in the bone window
  • Boundary: entire bone structure (typical shape was always recognizable in 3D view)
• Cricoid cartilage
  • Segment in the larynx window
  • Boundary: entire cartilaginous structure (typical shape usually visible in 3D view)
  • Special note: Caudal boundary simultaneously limits hypopharynx, larynx air, and inferior constrictor
• Pharyngeal constrictor muscles (superior, medius, inferior)
  • S. from the level of upper jaw teeth caudally
  • M. from hyoid cranially (both structures "meet" in the middle)
  • I. from hyoid caudally to the caudal end of the cricoid cartilage
• Esophagus
  • Cranial boundary: caudal end of the cricoid cartilage
  • Caudal boundary irrelevant for head-neck region, as the esophagus ends in the stomach
• C1/vertebral bodies
  • Segment in the bone window
• Soft palate
  • Cranial boundary: transition to hard palate
  • Caudal boundary: uvula
• Hard palate
  • Segment in the larynx window
• Larynx
  • Cranial boundary: epiglottis
  • Caudal boundary: caudal end of the cricoid cartilage
• Mandible
  • Segment in the bone window
  • Teeth not segmented
• Digastric muscle
  • Cranial boundary: mandible
  • Caudal boundary: until no longer visible
• Nasal cavities
  • Cranial boundary: until no longer visible
  • Caudal boundary: together with nasopharynx
  • Note: Exclude ethmoid cells
• Oral cavities
  • Includes tongue, uvula
• External Auditory Canal
• Tonsils
  • Bilateral at the level of uvula
• Common Carotid Artery
  • Cranial boundary: until artery bifurcation
  • Caudal boundary: branching from brachiocephalic trunk
• Sternal manubrium
  • Segment in the bone window
  • Note: Manubrium is posterior at transition with corpus sterni
• Sternum body
  • Segment in the bone window
  • Note: Corpus is anterior at transition with manubrium
• Clavicle
  • Segment in the bone window
• Zygomatic arch
  • Segment in the bone window
  • Ventral boundary: continuation from posterior edge of maxillary sinus
  • Dorsal boundary: up to mastoid cells
• Styloid process
  • Segment in the bone window
  • Cranial boundary: first slice where not connected to mastoid
  • Caudal boundary: until no longer visible
• Lung
  • Segment in the lung window
  • Often already exists
  • ’Region growing’ with upper threshold = -300 and ’remove holes’, but avoid including trachea/air outside the patient (sometimes segmented, correct manually)
• Trachea
  • Cranial boundary: larynx air
  • Caudal boundary: bifurcation
  • Excludes bronchi
• Internal Carotid Artery
  • Cranial boundary: entry into the skull
  • Caudal boundary: separation of common carotid
• Internal Jugular Vein
  • Cranial boundary: entry into the skull
  • Caudal boundary: brachiocephalic vein
• Trapezius muscles
  • Cranial boundary: skull
  • Caudal boundary: from the spine
  • Note: At the clavicle, trapezius also extends anteriorly, creating a tight "hole" in segmentation where tendon lies
• Platysma Muscle
  • Boundaries not clear but segmented as long as visible course toward mandible
• Brachiocephalic Artery
  • Cranial boundary: up to division into common carotids
  • Caudal boundary: from aortic arch
• Brachiocephalic vein
  • Cranial boundary: up to division into IJV
  • Caudal boundary: from SVC division
• Submandibular Gland
  • Segment as long as visible within platysma
• Levator Scapulae Muscle
  • Cranial boundary: as far as possible
  • Caudal boundary: from scapula
• Scalenus muscles (anterior, medius, posterior)
  • A. and M. around subclavian artery
  • A. and M. originate from first rib
  • P. often unclear, originates from second rib
  • All three structures traced cranially as far as possible
• Subclavian Artery
  • Lateral boundary: up to cranial boundary of sternum
• Skin
  • Adopt from outline or external contour and correct significant errors from automatic contouring
• Sterno-thyroid muscle
  • Cranial boundary: first slice where thyroid cartilage is ventrally united
  • Caudal boundary: first slice from manubrium
• Thyro-hyoid muscle
  • Cranial boundary: first slice where hyoid is visible
  • Caudal boundary: first slice after sternothyroid
• Pre-vertebral muscles (longus colli + longus capiti)
  • Cranial boundary: up to visible dens axis
  • Caudal boundary: as far as possible

Appendix A.2. Previously Reported DICE Values for Comparison

Table A1. Previously reported DICE values (mean ± standard deviation) between contours predicted by different deep learning methods and manual labels.

Structure	Previously reported DICE (mean ± std)
Mandible	0.86 ± 0.121[56], 0.90 ± 0.04 [54], 0.91 ± 0.02 [55], 0.94 ± 0.01 [52], 0.99 ± 0.01 [55]
Submandibular Gland (r)	0.73 ± 0.09 [54], 0.78 ± 0.10 [52], 0.79 [51], 0.95 ± 0.07 [55], 0.98 ± 0.03 [55]
Submandibular Gland (l)	0.70 ± 0.13 [54], 0.77 ± 0.12 [52], 0.79 [51], 0.91 ± 0.08 [55], 0.97 ± 0.05 [55]
Internal Carotid Artery (r)	0.81 [49], 0.86 ± 0.02 [50]
Internal Carotid Artery (l)	0.81 [49], 0.86 ± 0.02 [50]
Superior Constrictor	0.67 ± 0.11 [59], 0.76 ± 0.13 [55], 0.83 ± 0.15 [55]
Middle Constrictor	0.60 ± 0.19 [59], 0.76 ± 0.10 [55], 0.84 ± 0.01 [55]
Inferior Constrictor	0.65 ± 0.12 [59], 0.71 ± 0.21 [55], 0.80 ± 0.24 [55]
Constrictors (s., m., i.)	0.52 [51], 0.68 ± 0.08 [52]
Esophagus	0.85 ± 0.10 [55], 0.91 ± 0.03[52], 0.93 ± 0.07 [55]
Oral Cavity	0.85 ± 0.10 [55], 0.91 ± 0.03 [52], 0.93 ± 0.07 [55]

¹The values are only estimated from presented graphs in the referenced paper.

Table A1. Previously reported DICE values (mean ± standard deviation) between contours predicted by different deep learning methods and manual labels.

Structure	Previously reported DICE (mean ± std)
Mandible	0.86 ± 0.121[56], 0.90 ± 0.04 [54], 0.91 ± 0.02 [55], 0.94 ± 0.01 [52], 0.99 ± 0.01 [55]
Submandibular Gland (r)	0.73 ± 0.09 [54], 0.78 ± 0.10 [52], 0.79 [51], 0.95 ± 0.07 [55], 0.98 ± 0.03 [55]
Submandibular Gland (l)	0.70 ± 0.13 [54], 0.77 ± 0.12 [52], 0.79 [51], 0.91 ± 0.08 [55], 0.97 ± 0.05 [55]
Internal Carotid Artery (r)	0.81 [49], 0.86 ± 0.02 [50]
Internal Carotid Artery (l)	0.81 [49], 0.86 ± 0.02 [50]
Superior Constrictor	0.67 ± 0.11 [59], 0.76 ± 0.13 [55], 0.83 ± 0.15 [55]
Middle Constrictor	0.60 ± 0.19 [59], 0.76 ± 0.10 [55], 0.84 ± 0.01 [55]
Inferior Constrictor	0.65 ± 0.12 [59], 0.71 ± 0.21 [55], 0.80 ± 0.24 [55]
Constrictors (s., m., i.)	0.52 [51], 0.68 ± 0.08 [52]
Esophagus	0.85 ± 0.10 [55], 0.91 ± 0.03[52], 0.93 ± 0.07 [55]
Oral Cavity	0.85 ± 0.10 [55], 0.91 ± 0.03 [52], 0.93 ± 0.07 [55]

¹The values are only estimated from presented graphs in the referenced paper.