Application of an interactive diagnosis ranking algorithm in a simulated vignette-based environment for general dermatology
Keywords:
logic, diagnosis, algorithm, ranking, artificial intelligence, dermtrainer, knowledge base, teledermatology, human-computer interactionAbstract
Background: Diagnostic algorithms may reduce noise and bias and improve interrater agreement of clinical decisions. In a practical sense, algorithms may serve as alternatives to specialist consultations or decision support in store-and-forward teledermatology. It is, however, unknown how dermatologists interact with algorithms based on questionnaires.
Objective: To evaluate the performance of a questionnaire-based diagnostic algorithm when applied by users with different expertise.
Methods: We created 58 virtual test cases covering common dermatologic diseases and asked five raters with different expertise to complete a predefined clinical questionnaire, which served as input for a disease ranking algorithm. We compared the ranks of the correct diagnosis between users, analysed the similarity between inputs of different users, and explored the impact of different parts of the questionnaire on the final ranking.
Results: When applied by a board-certified dermatologist, the algorithm top-ranked the correct diagnosis in the majority of cases (median rank 1; IQR: 1.0; mean reciprocal rank 0.757). The median rank of the correct diagnosis was significantly lower when the algorithm was applied by four dermatology residents (median rank 2-5, p<.01 for all). The lowest similarity between inputs of the residents and the board-certified dermatologist was found for questions regarding morphology. Sensitivity analysis showed the highest deterioration in performance after omission of information on morphology and anatomic site.
Conclusions: A simple questionnaire-based disease ranking algorithm provides accurate ranking for a wide variety of dermatologic conditions. When applied in clinical practice, additional measures may be needed to ensure robustness of data entry for inexperienced users.
References
2. Verhoeven EWM, Kraaimaat FW, van de Kerkhof PCM, et al. Prevalence of physical symptoms of itch, pain and fatigue in patients with skin diseases in general practice. British Journal of Dermatology. 2007;156(6):1346-1349. doi:10.1111/j.1365-2133.2007.07916.x
3. Hay RJ, Johns NE, Williams HC, et al. The global burden of skin disease in 2010: an analysis of the prevalence and impact of skin conditions. J Invest Dermatol. 2014;134(6):1527-1534.
4. Hay RJ, Augustin M, Griffiths CEM, Sterry W, Board of the International League of Dermatological Societies and the Grand Challenges Consultation groups. The global challenge for skin health. Br J Dermatol. 2015;172(6):1469-1472.
5. Tschandl P, Rinner C, Apalla Z, et al. Human–computer collaboration for skin cancer recognition. Nat Med. 2020;26(8):1229-1234.
6. Polesie S, Gillstedt M, Kittler H, et al. Attitudes towards artificial intelligence within dermatology: an international online survey. Br J Dermatol. 2020;183(1):159-161.
7. Lallas A, Argenziano G. Artificial intelligence and melanoma diagnosis: ignoring human nature may lead to false predictions. Dermatol Pract Concept. 2018;8(4):249-251.
8. di Ruffano LF, Takwoingi Y, Dinnes J, et al. Computer‐assisted diagnosis techniques (dermoscopy and spectroscopy‐based) for diagnosing skin cancer in adults. Cochrane Database Syst Rev. 2018;(12). doi:10.1002/14651858.CD013186
9. Dick V, Sinz C, Mittlböck M, Kittler H, Tschandl P. Accuracy of Computer-Aided Diagnosis of Melanoma: A Meta-analysis. JAMA Dermatol. Published online June 19, 2019. doi:10.1001/jamadermatol.2019.1375
10. Ferrara G, Argenyi Z, Argenziano G, et al. The influence of clinical information in the histopathologic diagnosis of melanocytic skin neoplasms. PLoS One. 2009;4(4):e5375.
11. Cerroni L, Argenyi Z, Cerio R, et al. Influence of evaluation of clinical pictures on the histopathologic diagnosis of inflammatory skin disorders. Journal of the American Academy of Dermatology. 2010;63(4):647-652. doi:10.1016/j.jaad.2009.09.009
12. Binder M, Kittler H, Dreiseitl S, Ganster H, Wolff K, Pehamberger H. Computer-aided epiluminescence microscopy of pigmented skin lesions: the value of clinical data for the classification process. Melanoma Res. 2000;10(6):556-561.
13. Yap J, Yolland W, Tschandl P. Multimodal Skin Lesion Classification using Deep Learning. Exp Dermatol. Published online September 5, 2018. doi:10.1111/exd.13777
14. Liu Y, Jain A, Eng C, et al. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020;26(6):900-908.
15. Höhn J, Hekler A, Krieghoff-Henning E, et al. Integrating Patient Data Into Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review. J Med Internet Res. 2021;23(7):e20708.
16. Pacheco AGC, Krohling R. An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification. IEEE J Biomed Health Inform. 2021;PP. doi:10.1109/JBHI.2021.3062002
17. Tognetti L, Bonechi S, Andreini P, et al. A new deep learning approach integrated with clinical data for the dermoscopic differentiation of early melanomas from atypical nevi. J Dermatol Sci. 2021;101(2):115-122.
18. Chou W-Y, Tien P-T, Lin F-Y, Chiu P-C. Application of visually based, computerised diagnostic decision support system in dermatological medical education: a pilot study. Postgrad Med J. 2017;93(1099):256-259.
19. Holubar K. Ferdinand von Hebra 1816--1880: on the occasion of the centenary of his death. Int J Dermatol. 1981;20(4):291-295.
20. Salzer G, Ciabattoni A, Fermüller C, et al. Dermtrainer: A Decision Support System for Dermatological Diseases. arXiv [csIR]. Published online July 1, 2019. http://arxiv.org/abs/1907.00635
21. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2019. https://www.R-project.org/
22. Holm S. A Simple Sequentially Rejective Multiple Test Procedure. Scand Stat Theory Appl. 1979;6(2):65-70.
23. Wickham H. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York; 2016.
24. Ceney A, Tolond S, Glowinski A, Marks B, Swift S, Palser T. Accuracy of online symptom checkers and the potential impact on service utilisation. PLoS One. 2021;16(7):e0254088.
25. Gilbert S, Mehl A, Baluch A, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open. 2020;10(12):e040269.
26. Burke MD, Savard LB, Rubin AS, Littenberg B. Barriers and facilitators to use of a clinical evidence technology in the management of skin problems in primary care: insights from mixed methods. J Med Libr Assoc. 2020;108(3):428-439.
27. Groh M, Harris C, Soenksen L, et al. Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. ; 2021:1820-1828.
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Antonia Wesinger, Elisabeth Riedl, Harald Kittler, Philipp Tschandl
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Dermatology Practical & Conceptual applies a Creative Commons Attribution License (CCAL) to all works we publish (http://creativecommons.org/licenses/by-nc/4.0/). Authors retain the copyright for their published work.
