Expert agreement on the presence and spatial localization of melanocytic features in dermoscopy

: Dermoscopy aids in melanoma detection; however, agreement on dermoscopic features, including those of high clinical relevance, remains poor. Herein we attempted to evaluate agreement among experts on 'exemplar images' not only for the presence of melanocytic-specific features but also for spatial localization. This was a cross-sectional, multicenter, observational study. Dermoscopy images exhibiting at least one of 31 melanocytic-specific features were submitted by 25 world experts as 'exemplars'. Using a web-based platform that allows for image mark-up of specific contrast-defined regions (superpixels), 20 expert readers annotated 248 dermoscopic images in collections of 62 images. Each collection was reviewed by five independent readers. A total of 4,507 feature observations were performed. Good-to-excellent agreement was found for 14 of 31 features (45.2%), with 8 achieving excellent agreement (Gwet's AC >0.75) and 7 of them being melanoma-specific features. These features were: 'Peppering /Granularity' (0.91); 'Shiny White Streaks' (0.89); 'Typical Pigment network' (0.83); 'Blotch Irregular' (0.82); 'Negative Network' (0.81); 'Irregular Globules' (0.78); 'Dotted Vessels' (0.77) and 'Blue Whitish Veil' (0.76). By utilizing an exemplar dataset, good-to-excellent agreement was found for 14 features that have previously been shown useful in discriminating nevi from melanoma. All images are public (www.isic-archive.com) and can be used for education, scientific communication and machine learning experiments.