Inter-rater Agreement for Diagnosing Adenomyosis Using Magnetic Resonance Imaging and Transvaginal Ultrasonography

1. Introduction

Adenomyosis is a benign uterine disease characterized by the presence of ectopic endometrial glands and stroma within the myometrial layer, surrounded by an area of hypertrophic myometrium [1]. Symptomatology of adenomyosis includes heavy menstrual bleeding and pelvic pain [2-4]. Traditionally, the diagnosis of adenomyosis was only possible through histological examination after hysterectomy. However, advancements in medical imaging techniques, specifically Magnetic Resonance Imaging (MRI) and Transvaginal Ultrasonography (TVS), have enabled non-invasive diagnosis of adenomyosis

MRI has been used as a diagnostic tool for adenomyosis, with the increased thickness of the Junctional Zone (JZ) with a suggested cut-off of 12mm [5], and the presence of myometrial cysts being a common criteria[6]. The principal limitation of MRI is the absence of a definable junctional zone on imaging, which occurs in 20% of premenopausal women [5].

Later studies have suggested additional features, such as an irregular appearance of the JZ and its relationship with the thickness of the entire myometrium [7-10]. Another diagnostic feature is presence of small punctate cystic foci located within the JZ [7, 11]. The diagnosis of adenomyosis by MRI is highly reproducible [9, 12] with high inter-rater agreement for various JZ measurements [9].

TVS, on the other hand, utilizes two-dimensional (2D) and three-dimensional (3D) imaging techniques to assess adenomyosis. The sonographic features used for diagnosing adenomyosis are well describe using TVS [13-16]. A recent consensus paper by the MUSA (Morphological Uterus Sonographic Assessment) consortium distinguish between direct features of adenomyosis i.e., myometrial cysts, hyperechogenic islands and echogenic subendometrial lines and buds, indicating presence of ectopic endometrial tissue in the myometrium, and indirect features, features reflecting changes in the myometrium secondary to presence of endometrial tissue in the myometrium, i.e. globular uterus, asymmetrical myometrial thickening, fan-shaped shadowing, translesional vascularity, irregular junctional zone and interrupted junctional zone [17]. The JZ is poorly visualized with 2D TVS. Three dimensional (3D) TVS offer the possibility to assess the coronal plane and to use VCI (Volume Contrast Imaging, i.e. thick slice) to improve visualization of the JZ which facilitates evaluation of thickness and irregularity [14, 18, 19]. Reproducibility studies have shown a high level of agreement when evaluating the presence or absence of adenomyosis using TVS [17, 20-22], but less agreement when assessing different ultrasound features of the disease [20, 21].

The aim of this study was to compare the inter-rater agreement of TVS with MRI regarding diagnosing adenomyosis and regarding the assessment of predefined imaging features. By assessing both modalities in the same set of women, we aim to provide insights into their respective diagnostic capabilities and agreement levels.

2. Materials and Methods

2.1. Study Design and Eligibility Criteria

Women aged 35-50 years with heavy regular menstrual bleeding and suspected adenomyosis on clinical examination (including TVS) were recruited in a private gynecologic clinic in Stockholm between January 2014 and December 2016 and referred to MRI and expert TVS as part of the investigation. All women were clinically examined and recruited by the same gynecologist (JA). Women with adenomyoma, known endometriosis, uterine leiomyomas >4cm, multiple leiomyomas (>3) or current use of intrauterine device or hormonal contraception were not eligible for the study.

In total 67 women were recruited and referred to MRI and TVS. MRI was missing in 3 women (declined examination, n=2; wrong identification number, n=1) and TVS in 13 women (images were not pseudonymized, n=2; 3D volumes, n=6; or 2D video sequences n=5 were not recorded or of insufficient quality). Both MRI and TVS images were available in 51 women and included in the study. MRIs were assessed by four experienced radiologists and pre-recorded 2D TVS videoclips and 3D volumes were assessed by five experienced gynecological ultrasonographers. Assessment of the images was performed between December 2019 and February 2020. The raters were blinded to the clinical history, physical examination and the evaluation of MRI and TVS images made by other raters.

2.2. MRI Examination and Assessment

MRI of the pelvis was performed in an outpatient facility on a 1.5T system (Optima MR450w, GE Healthcare, Waukesha, WI, USA or Siemens Magnetom Symphony Tim, Siemens Healthineers, Erlangen, Germany). The minimum protocol included the following sequences: T2-weigthed Fast Relaxation Fast Spin Echo (FRFSE) or Turbo Spin Echo (TSE) in the axial, sagittal and coronal plane (slice thickness 4-5 mm; gap: 10-20%); T1-weigthed Fast Spin Echo (FSE) or a Gradient Echo (GRE) in the axial and coronal plane (slice thickness 5 mm; gap 10-20%). All examinations were performed with a phased array coil. The women were asked to fast for 4 hours before the examination. Antispasmodic drugs were not administered.

The images were pseudonymized and evaluated on a Picture Archiving and Communication System (PACS; IDS7 version 21.1, Sectra AB, Linköping, Sweden) at the Karolinska University Hospital by four experienced radiologists. Raters could save their assessments and resume at a later time, to reduce risk of fatigue. With regard to the presence or absence of adenomyosis, each rater based the assessment on their subjective evaluation of the radiological features. There were no standardized criteria given on when to make the diagnosis. The predefined MRI features assessed are listed in Table 1 and shown in Figure 1.

2.3. TVS Examination and Assessment

All women underwent ultrasound examination by a single expert examiner (EE) at the Karolinska University Hospital using a high-end ultrasound system Voluson E10 or E8, GE Healthcare (GE Medical Systems, Zipf, Austria) with a 5-9 MHz 3D transvaginal probe. 2D grayscale, videoclips and 3D-VCI grayscale volumes including the whole uterine body were saved. The GE 4D View software (GE Healthcare, Wood Dale, IL, USA) was used to assess the 3D-VCI volumes. The pseudonymized video-clips and 3D volumes for each case were downloaded to memory sticks and sent to five experienced ultrasonographers. The raters used their own personal computers to assess the volumes. They were encouraged to use high-resolution computer screens and to perform the assessments in a dark room to avoid glare on the screen. The raters could save their assessments and resume later, to reduce risk of fatigue. The volumes were saved in the VCI format with a 2 mm thickness and with a grey mix of 70% X-ray/30% surface smooth. The raters could modify the volume during the analysis to optimize the assessment (remove the VCI function, change slice thickness or grey mix, rotate the volume in any plane). With regard to the presence or absence of adenomyosis, each rater based the assessment on their subjective evaluation of different ultrasonographic features, using pattern recognition [13]. The measurement of the anterior and posterior wall thickness was done in a longitudinal plane and the measurement of the junctional zone (JZ max and JZ min) in the reconstructed coronal plane from the 3D volume. The predetermined TVS features assessed are listed in Table 2 and shown in Figure 2a and 2b.

2.4. Statistical Analysis

Each rater entered the assessments into the Research Electronic Data Capture (REDCap^®, Vanderbilt University) data entry and management program [23, 24] hosted at the Karolinska Institutet. REDCap is a secure, web-based software platform designed to support data capture for research studies, providing automated export procedures for data downloads to common statistical packages. Statistical Data Analysis was performed using the Software (SPSS), version 26, IBM Corporation, Armonk, NY, USA.

Fleiss kappa (κ) was used as a measure of inter-rater agreement for categorical data. Kappa values were categorized as ‘Poor’ (κ≤0.20), ‘Fair’(0.21-0.40), ‘Moderate’ (0.41-0.60), ‘Good’(0.61-0.80), and ‘Very good’(0.81-1.00) [25].

The Intraclass correlation coefficient (ICC) was used as a measure of reliability for quantitative data, categorized as ‘Poor’ (≤0.5), ‘Moderate’ (0.5-0.75), ‘Good’ (0.75-0.9) and ‘Excellent’ (>0.9) and their 95% confident intervals were calculated based on individual-rating, absolute-agreement, 2-way random effects model [26, 27].

3. Results

The inter-rater agreement for diagnosing adenomyosis and for the individual imaging features are presented in Table 3 and 4.

The five raters of ultrasound images classified adenomyosis as present in 51%, 49%, 39%, 37% and 76% of the cases and the four raters of MRI classified the disease as present in 88%, 43%, 49% and 61% respectively.

The inter-rater agreement (κ=, [95% CI]) for diagnosis of adenomyosis was higher for TVS (κ =0.42, [.417-.422]) than for MRI (κ =0.28, [.28-.287]). For MRI the inter-rater agreement was moderate for wall asymmetry and irregular JZ and poor for globular uterine shape. For TVS the inter-rater agreement was moderate for wall asymmetry and globular uterine shape, fair for irregular JZ, fan-shaped shadowing, subendometrial buds and lines and hyperechogenic islands.

The assessment of cysts in the JZ/myometrium had fair agreement in MRI and cysts in the myometrium had poor agreement in TVS (κ =.327, [.324-.331] vs. κ =.192, [.19-.195]).

The reliability for JZmax was good (ICC=.821, [.709-.898]) for MRI. For TVS, the JZmax was classified as “not assessable” by one or more rater in 29/51 cases and ICC was unreliable in the remaining 22 cases (ICC=.082, [-0.11-.252]).

For other continuous data assessed with MRI, ICC showed good reliability for myometrial thickness and JZ differential (JZdiff) and a moderate reliability for JZmin and Ratio JZmax/Myometrium.

The raters of MRIs considered the quality of the images to be medium to high in 47%, 90%, 88% and 69% of the cases, while raters of TVS volumes and video clips considered medium to high quality in 82%, 86%, 67%, 76% and 88% of the cases (Table 5). In 71%, 94%, 98%, and 80% of the MRI cases and 96%, 94%, 74%, 80% and 78% of the TVS cases the raters were medium or highly confident in their evaluation (Table 5).

4. Discussion

We found that the inter-rater agreement for diagnosing adenomyosis was higher for TVS than for MRI, despite the inter-rater agreement for most individual imaging features being higher for MRI than for TVS. Moreover, MRI showed a clearly higher reliability than TVS for continuous variables. Since MRI had a higher agreement for most individual images features that were assessed by both TVS and MRI it is remarkable that the inter-rater agreement for diagnosing adenomyosis was lower for MRI than for TVS. Lack of standardized criteria for diagnosing adenomyosis for MRI is the most likely reason. Since ICC for JZmax was good, the agreement for diagnosing adenomyosis with MRI may have improved if a cut off for JZ max had been used as criterion. However, using the JZ as the only criteria for adenomyosis is questioned [9, 10]. Also, the absence of cysts may have affected the result. Intramyometrial cysts are pathognomonic for the disease but are found in only one third to half of affected women [7, 8, 10]. Some of the participating radiologists may have excluded the diagnosis when no cysts were found, even when altered JZ features were present. The limited agreement for cysts may negatively have affected the agreement for diagnosis. A recent meta-analysis and review on the performance of various objective criteria diagnosing adenomyosis, using MRI, conclude that most parameters have a relatively low sensitivity and a relatively high specificity [28]. JZ characteristics remain the most widely used and investigated with acceptable diagnostic accuracy. Specific research is needed into how these objective measures of adenomyosis can be correlated to clinical outcomes.

For TVS, the inter-rater agreement was fair or moderate for most of the individual ultrasound features, and moderate for diagnosing adenomyosis. This is in line with results of other studies [18, 20] showing a good reproducibility diagnosing adenomyosis using 2D TVS pattern recognition. When an ultrasound diagnosis of adenomyosis is made using pattern recognition, all the different ultrasound features are taken into account. This may explain why agreement regarding diagnosis was higher than agreement for most individual features. When combined, the subjective overall agreement if adenomyosis is present or not becomes higher than the agreement of various variables. The inter-rater agreement for cysts in the myometrium was poor. Small cysts in the myometrium may be difficult to detect with TVS, especially when other features, such as shadowing, is present. In this study, globular uterus shape and wall asymmetry showed the highest agreement between raters, whereas other studies have reported the highest agreement for irregular JZ [20, 21].

With 3D TVS, it is possible to visualize the JZ. However, JZ measurements with 3D TVS has been shown to have limited reproducibility [20-22]. In our study, there was missing data from >50% of the TVS cases, since cases were excluded from the ICC analysis if one rater classified the JZ as “not assessable”, even though the other raters measured the JZ. Moreover, results were unreliable in the cases where all the TVS raters measured the JZ, indicating low diagnostic value in clinical practice. One reason may be low quality of 3D volumes in women with adenomyosis, as the abundant scattering of the ultrasound beam results in poor image quality in the reconstructed coronal plane, making assessment difficult. Moreover, the coronal plane may not be the correct plane to measure JZ. A recent update on the MUSA consensus paper suggest to assess the regularity of the junctional zone in multiple planes (transversal, longitudinal, coronal) using 3D ultrasound, since a regular JZ can rule out adenomyosis, while measurements of the JZ was dismissed because of lack of evidence of the clinical relevance of this measurement [30].

A recent meta-analysis showed that MRI and TVS had adequate performance with regard to diagnosing adenomyosis, with a pooled sensitivity and specificity of 75% and 81% for TVS and 69% and 80% for MRI, p=0.75, when hysterectomy was used as gold standard [29]. Still, both modalities have shown unsatisfactory inter-rater agreement for the diagnosis, indicating that adenomyosis is challenging even for experts. Clear definitions and criteria along with more standardized interpretation models for imaging are needed for both MRI and TVS to make the diagnosis more reliable.

A strength of the study is the large number of cases examined with both imaging modalities and the enrollment of raters from different centers. For TVS, expert raters from different centers in Europe were included, which increases the generalizability of the study. Unfortunately, this was not possible for MRI.

Despite the TVS images were recorded off-line and a dedicated "state-of-the-art" MRI was not applied, the majority of raters considered the quality of the images medium or good and were confident in their evaluation. Although suboptimal image quality, it still appears acceptable for assessing the required parameters and reflects the reality in a clinical set-up.

The use of the RedCap database reduced human error in the handling of data to a minimum and ensured complete datasets. All questions were mandatory and thus had to be answered before moving to the next case. There is “missing data” only for the JZ measurements, where it was possible to classify the JZ as “not assessable”.

A limitation of the study is the use of stored off-line TVS videos and volumes instead of real-time examinations. However, it would be impossible to do reproducibility studies with multiple raters in TVS without using recorded material. Even for MRI, the quality of the images was not optimal. The examinations were performed in a private radiological outpatient center where neither abdominal belt nor antispasmodic agent was used to reduce motion artefacts caused by small bowel peristalsis. Several MRIs were affected by artefacts, thus hampering the quality of the images. However, consensus guidelines suggesting technical protocols for MR imaging of endometriosis were published after the enrollment of the study subjects [31, 32]. Furthermore, no oblique axial T2-weighted sequence perpendicular to the long uterine axis was included in the MRI protocol, which is useful for the assessment of adenomyosis [8, 10]. If cases with adenomyosis confirmed by histology and images with excellent quality had been chosen, the results may have been different. Since the study investigated the agreement between raters it is not necessary to have a gold standard for comparison. The current set-up gives information on a real-life setting.

5. Conclusions

The inter-rater agreement for diagnosing adenomyosis was higher for TVS than for MRI despite MRI manifesting higher inter-rater agreement in most variables, in particular variables related to measuring the junctional zone. Measurement of JZ thickness in the coronal plane with 3D TVS could only be done in less than half of the women and was found to be unreliable in the rest, and is therefore unlikely to be useful for diagnosing adenomyosis.