The level of agreement between self-assessments and expert assessments of melanocytic nevus counts: Findings from an evaluation of 4548 blinded double assessments

1. Introduction

Incidence rates of cutaneous melanoma (CM) have shown a dynamic development in countries with predominantly fair-skinned populations in recent decades [1]. In Scandinavian countries and New Zealand, where population-based cancer registries have documented trends over a long period since the 1940s, incidence rates have increased 30- to 40-fold over the last 70 years, with annual increases of 3-7% [1,2,3]. CM, which was once a rare form of skin cancer until the middle of the last century, is currently an important public health issue in many countries worldwide.

CM is an obvious candidate for screening programs, since its prognosis is excellent when it is diagnosed in an early stage of the disease [4,5,6]. Current evidence on the benefits of CM screening, summarized for the US Preventive Services Task Force in a recent systematic review covering 20 studies [7], is, however, inconsistent. Due to the high financial costs of population-wide screening for CM and the potential for overdiagnosis, i.e. the identification of otherwise indolent melanoma or false-positive melanoma diagnoses that turn out to be benign lesions [8], most countries have not issued a recommendation regarding CM screening thus far [9]. Only two countries have established national guidance on CM screening by physicians. Germany recommends visual skin examinations by dermatologists or general practitioners, who have been certified for performing skin cancer screening. Examinations can take place every two years (beginning at age 35) and are covered by statutory health insurance [10]. France recommends that general practitioners assess the CM risk of their patients via a checklist consisting of seven questions and refer high-risk patients for further diagnostic clarification to a dermatologist [11].

An alternative to population-wide screening is targeted screening of those at high risk for developing CM [12]. An efficient implementation of such a strategy requires that the CM risk can be assessed simply, reliably and validly. To this end, the number of melanocytic nevi has been consistently identified as a strong predictor of CM [13,14]. Nearly all prediction models developed to date to capture CM risk have incorporated the number of nevi, mostly in a categorized form, among the explanatory variables [15]. A prerequisite for using the number of nevi in order to identify individuals at high CM risk for targeted screening is, however, that this variable can be validly self-assessed. Otherwise physicians would need to be involved in identifying the high-risk group for actual screening, and potential savings compared to population-wide screening could not be achieved.

By addressing the aspect of validity when using self-assessment to identify the number of melanocytic nevi, the objective of our study was to analyze the level of agreement between self-assessment and expert assessment of nevus counts based on a large dataset of study subjects in whom mutually blinded double counting of nevi was performed. We report on the quantitative degree of agreement between the two assessments and on potential determinants of the agreement.

2. Materials and Methods

2.1. Design and Participants

The study started in April 2006 as a repeated cross-sectional survey and was conducted in an identical way twice a year (in April and November) until November 2019 at the University of Erlangen-Nuremberg. Every summer and winter semester, all students enrolled in the course ‘Biometry and Epidemiology’ were invited to participate in the study. Of the 5125 students enrolled during the 13.5-year study period, 4768 students attended the course on the day when the study took place. Of these, 4704 (98.7%) agreed to participate and provided written informed consent. The study was approved by the Ethics Committee of the University Erlangen-Nuremberg.

2.2. Nevus Counting

The criteria for the nevus counting procedure followed the IARC protocol for identifying and recording nevi [16]. Accordingly, all common acquired melanocytic nevi with a diameter of at least 2 mm were eligible for counting. For feasibility reasons, nevus counting was limited to the area between the wrist and the shoulder, as nevus counts on the arms are highly correlated with total body counts [17]. The number of melanocytic nevi on both arms was counted twice, once by the participant and once by one of six specifically trained examiners involved in the study (one dermatologist, two physicians of other disciplines, and three academic researchers who were instructed by the dermatologist), in a mutually blinded manner: participants were not aware of the counting result of the examiner when performing the nevus counting and vice versa (for logistical reasons, the sequence of nevus counting was not identical for all participants, in one subgroup examiners counted first, in the other subgroup examiners counted last).

Participants received detailed written instructions and photographs of various skin lesions to clarify the counting procedure and appearance of melanocytic nevi and to distinguish them from other skin lesions such as café au lait spots, Becker’s nevus, viral warts, freckles, hemangiomas, dermatofibromas, and other potentially similar lesions. Additionally, they were given a mirror to facilitate counting on the dorsal upper areas of the arms and a “nevometer” [18], a small, 2-mm thick transparent polymethylmethacrylate spatula with a hole of 2 mm in diameter, to be used as reference when deciding whether the size of a lesion was above the 2 mm threshold.

Standardization of the nevus counting procedure between the examiners who performed the expert assessments was essential for the study. To achieve this goal, regular training sessions were held before the study dates in each winter and summer semester, during which the examiners discussed all aspects of the nevus counting procedure and carried out practical exercises.

2.3. Questionnaire

In addition to nevus counting, information from the study participants was collected using a self-administered questionnaire that comprised sociodemographic variables (age, sex, degree course) and phenotype information (hair and eye color, freckling, skin type). Participants were provided with an eye and hair color reference chart, an illustration of different categories of freckling, and a detailed description of how to assess the Fitzpatrick skin type [19] in order to standardize the assessment of phenotype.

2.4. Statistical Analysis

Agreement between the two nevus assessments was analyzed based on (i) the raw results of the nevus counting and (ii) a categorized nevus score (categories defined by the quintiles of the nevus distribution). For these analyses all participants with Fitzpatrick skin types V and VI were excluded. For the raw nevus counts, the mean difference (± standard deviation (SD)) between the two assessments and Pearson’s correlation coefficient, accompanied by its 95%-confidence interval (CI), were computed to quantify the relationship between the two nevus asessments. Additionally, the differences between the two nevus counting results were examined graphically using a Bland–Altman plot, which displays the difference in relation to the average count as a surrogate for the unknown true number of nevi. For the categorized nevus score, raw agreement (in percent) and chance-corrected agreement by Cohen’s weighted kappa coefficient (κ) were estimated from the 5x5-contingency table showing the joint distribution of participants’ and experts’ nevus classification [20]. The precision of these estimates is depicted by 95%-CIs.

To evaluate potential determinants of agreement, subgroups of the total study group were defined by the variables sex (male/female), degree course (clinical medicine/other), semester (summer/winter), skin type (Fitzpatrick type I–IV), freckles (yes/no), hair color (red, blonde, brown, black), and eye color (dark blue, light blue, green, green-brown, light brown, dark brown). Weighted kappa coefficients were estimated in the subgroups and differences between subgroup-specific estimates were statistically evaluated for each of the seven variables by the test for equal kappa coefficients between independent groups developed by Fleiss et al. [21]. P-values less than 0.05 were considered to indicate statistical significance but should be interpreted in an explorative manner as no adjustment for multiple testing was made. All statistical analyses were carried out using R software version 4.3.2 (R Foundation for Statistical Computing, Vienna, Austria). In particular, the R package irrCAC was used for estimating weighted kappa coefficients.

3. Results

Of the recruited study group (n=4704), 90 participants were excluded from the analysis due to their skin type (i.e., Fitzpatrick type V–VI), 48 had incomplete information about their nevus status because one of the assessments was missing, and data from additional 18 participants with double counting results were excluded from the analysis because the expert assessment was carried out by a one-time substitute examiner. The remaining 4548 participants comprised 1689 (37%) men, 2845 (63%) women and 14 (0.3%) participants who did not reveal their gender. The participants’ mean (± SD) age was 23.53 (± 3.36) years. Altogether, 4156 (91%) of the participants studied clinical medicine, and 392 (9%) came from other disciplines (molecular medicine, life science engineering, speech therapy). Nearly equal numbers of participants completed the study during the summer term (n= 2241, 49%) and the winter term (n=2307, 51%). A descriptive summary of the distribution of phenotype variables in the study sample is given in Table 1.

3.1. Distribution of Nevus Counts

Both distributions, that of the self-assessed number of nevi and that of the expert-assessed number of nevi, were strongly skewed. While the median number of self-assessed (expert-assessed) nevi was 14 nevi (12 nevi), the arithmetic mean was 18.9 nevi (15.6 nevi). When defining the nevus score, i.e., the categorized version of the counted number of nevi, we used the joint distribution of all nevus counts from participants and experts. The quintiles of this joint distribution were 5, 10, 16, and 26 nevi, leading to the following five categories for the nevus score: ≤5, 5-10, 10-16, 16-26, and >26.

3.2. Nevus Counts: Differences Between Assessments

The results of self-assessed and expert-assessed nevus counting showed a correlation of 0.774 (95%-CI: 0.762 – 0.786). The self-assessed nevus counts were somewhat greater than the expert-assessed counts, and the mean difference (± SD) in the total sample was 3.33 (± 11.29). Figure 1 displays the Bland-Altman plot for the data showing the individual mean differences in relation to the averages of the two counts.

3.3. Nevus Score: Agreement Between Assessments

The joint distribution of the self-assessed and expert-assessed nevus scores is shown in Table 2. Based on the five-level score, the raw agreement in the total sample was 50.29% (95%-CI: 48.83% - 51.74%). The chance-corrected agreement was moderate (κ=0.596, 95%-CI: 0.581-0.611). Sensitivity analyses using a categorized nevus score with a different number of categories between 3 and 7 (using a data-adaptive definition of the specific categories, i.e. tertiles, quartiles, etc.) yielded stable results regarding estimated weighted kappa coefficients.

Detailed results of the agreement between the two assessments in the subgroups defined by sociodemographic and phenotype variables are shown in Table 3. Overall, the level of agreement was quite homogenous across the subgroups. For all variables, except eye color, we found nonsignificant variation of limited magnitude in the weighted kappa estimates of the corresponding subgroups. However, in the subgroups defined by participants’ eye color, we observed a much lower weighted kappa of 0.512 (95%-CI: 0.462-0.561) for those with dark blue eyes and a higher weighted kappa of 0.642 (95%-CI: 0.598-0.685) for those with dark brown eyes, whereas the other four subgroups showed weighted kappas at the same level as in the total sample. Heterogeneity in the level of agreement between the six subgroups defined by participants’ eye color reached significance (p=0.01).

4. Discussion

In a controlled study setting, we evaluated nevus self-counting by comparing self-assessed and expert-assessed nevus counts obtained in a mutually blinded fashion. Based on 4548 double counting results we observed only a moderate level of agreement between the two assessments. Self-assessed nevus counts were higher than those determined by experts. The level of agreement was independent of sociodemographic and cutaneous factors. We found, however, a greater than average level of agreement in participants with dark brown eyes and a lower than average level of agreement in those with dark blue eyes. Participants’ eye color was the only factor that had a significant impact on the level of agreement in our study.

Tackling aspects of validity and reliability of nevus self-counting is not a new topic in dermato-epidemiologic research. In a systematic literature search, we identified 21 studies published between 1991 and 2021 [18,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41] that reported some data on the relationship between self-counting and expert counting of nevi on the same individuals; 13 of these studies focused primarily on this topic. All previous investigations were based on much smaller study samples comprising 46 to 1772 participants which limits the precision of quantifying the level of agreement. Most of them were embedded in studies with melanoma patients or used participants of screening programs for skin cancer as their study subjects. Unsurprisingly, the results of these studies were very heterogeneous as the investigations differed not only with respect to the origin and size of the study population but also with respect to the instructions for self-counting given to study participants. The lowest level of agreement was observed in a study of 1658 employees of bank and insurance companies who had only some minutes to fill out a self-administered questionnaire prior to a voluntary skin check by a dermatologist [39]. The questionnaire comprised one question asking participants to estimate the total number of moles on their body (0-10, 11-30, 31-50, 51-100, >100) without further explanation and instructions. These self-assessments of nevi showed virtually no agreement beyond chance with the results of nevus counting by the dermatologist (weighted κ=0.03). Other studies have reported higher levels of agreement, which is more consistent with our findings. For example, Jackson et al.’s study [26] in the United Kingdom, comprising eight general practitioner practices with 388 participants, found an identical level of agreement (κ=0.60) as our study for a three-level nevus score. A study by Mannino et al. [41] in three European dermatologic clinics that assisted participants in assessing their nevi with instructions and photographs similar to ours reported a moderate level of agreement (weighted κ=0.45), slightly below our result, in their sample of 744 patients.

The vast majority (91%) in our study sample were students attending a compulsory course in a medical degree program. Of course, such a sample is not representative of the general population. At the time of the study, however, the students had no previous dermatologic training and thus had no experience in distinguishing melanocytic nevi from other skin lesions. Nevertheless, we cannot exclude the possibility that their general medical training and attitudes toward this topic may have influenced their motivation and diligence when participating in the study, which may have biased the results toward a higher level of agreement in our study than would be expected in the general population.

When self-assessed information on melanoma risk factors is used to identify a subgroup in the population at high risk of developing melanoma as a target of screening activities, the self-assessed information needs to be accurate. Although prediction models developed for estimating melanoma risk usually include several variables, the frequency of nevi is a particularly important component of such models [15]. From the literature on melanoma prediction models there is some empirical evidence that the discriminatory properties of melanoma prediction models incorporating self-assessed nevus information are inferior to those using physician-assessed nevus information. Four publications [42,43,44,45] described melanoma prediction models derived from the same data, namely, the Australian Melanoma Family Study [46], and validated them using data from, again, the same population-based case-control study in the United Kingdom [47]. The prediction models differed with respect to the incorporated variables: two [44,46] used genotype information in addition to phenotype and UV exposure variables, while the remaining two [43,45] focused on nongenetic risk factors. The difference between the latter two models related to the incorporation of only self-assessed phenotype variables in [43] and the use of physician-assessed phenotype variables in [45]. The model performance differed significantly, the AUC parameter describing model discrimination ranged from 0.66 for the model including only self-assessed risk factors without genotype information to 0.79 for the model including physician-assessed phenotype variables and genotype information. The main driver of the increase in the AUC was the use of physician-assessed nevus counts instead of self-assessed nevus information, while the use of genotype information had only a modest impact on the AUC.

Our large-scale investigation showed that even detailed instructions including photographs explaining the definition of melanocytic nevi, i.e., which skin lesions should and should not be counted, failed to increase the level of agreement between self-assessed and expert-assessed nevus counts to an acceptable level of substantial or near-perfect agreement. Together with the results of other studies it seems questionable to use self-assessed information on phenotype variables such as the number of nevi when defining a high-risk subgroup of the population for targeted melanoma screening.

5. Conclusions

Our results show that even with comprehensive guidance, only a moderate level of agreement between self-assessed and expert-assessed nevus counts can be achieved. Self-assessed nevus information does not appear to be reliable enough to be used in individual risk assessment for targeted screening programs.