Artificial Intelligence in Pathology
; ; ;
Background: Increasing digitalization enables the use of artificial intelligence (AI) and machine learning in pathology. However, these technologies have only just begun to be implemented, and no randomized prospective trials have yet shown a benefit of AI-based diagnosis. In this review, we present current concepts, illustrate them with examples from representative publications, and discuss the possibilities and limitations of their use.
Methods: This article is based on the results of a search in PubMed for articles published between January 1950 and January 2020 containing the searching terms “artificial intelligence,” “deep learning,” and “digital pathology,” as well as the authors’ own research findings.
Results: Current research on AI in pathology focuses on supporting routine diagnosis and on prognostication, particularly for patients with cancer. Initial data indicate that pathologists can arrive at a diagnosis faster and more accurately with the aid of a computer. In a pilot study on the diagnosis of breast cancer, involving 70 patients, sensitivity for the detection of micrometastases rose from 83.3% (by a pathologist alone) to 91.2% (by a pathologist combined with a computer algorithm). The evidence likewise suggests that AI applied to histomorphological properties of cells during microscopy may enable the inference of certain genetic properties, such as mutations in key genes and deoxyribonucleic acid (DNA) methylation profiles.
Conclusion: Initial proof-of-concept studies for AI in pathology are now available. Randomized, prospective studies are now needed so that these early findings can be confirmed or falsified.
Being concerned with the histomorphological analysis of human tissue specimens, the discipline of pathology plays a key role in the diagnostic workup. For example, guideline-adherent management of numerous oncological conditions requires prior histopathological confirmation of the diagnosis and the number of drugs requiring molecular pathological identification of predictive biomarkers prior to their use is constantly increasing (1, 2). In their daily routine, pathologists generate, analyze and integrate large amounts of data coming from various sources: extensive clinical information, image data from histological and immunohistochemical stainings or molecular pathological data from sequence analyses. The rapid development of whole slide scanning over the last two decades has enabled the digitalization of typically analog microscopic image information in sufficient quality and quantity (3). However, while other imaging disciplines have adopted a largely computer-based work style for years now, digital transformation has only just started in pathology (4). One of the most promising developments in this respect could be the utilization of artificial intelligence (AI) und machine learning (ML) (5). However, AI- and ML-based technologies have not yet been evaluated in a prospective, randomized trial and their adoption in routine pathology is far from being widespread. Thus, the research discussed below should be regarded as proof-of-concept/pilot studies, evaluating various aspects in pathology pertaining to specific AI procedures and identifying potential future areas of application (Table 1). While these are representative for the clinical questions addressed, they have not yet been replicated. This review is based on pertinent publications retrieved by a selective search in the PubMed database for the period from January 1950 to January 2020, using the search terms “artificial intelligence“, “deep learning“ and “digital pathology“, as well as the results of our own research.
Artificial intelligence and digital image analysis
AI describes the process of teaching a machine to solve a problem without the need to explicitly specify each step to the solution. AI research and development is a subfield of computer science. Machine learning, in turn, is concerned with specific methods that can be used to implement AI. Here, a distinction is sometimes made between classical machine learning and newer approaches, such as deep learning. A number of developments in the last decade have led to a steep increase in AI applications: improved algorithms, improved hardware and a rapidly growing amount of obtainable data. These developments paralleled the trend in pathology towards increasing digitalization (Figure 1).
These key developments have occurred mostly in the field of computer-based, automated processing of image data, which is also referred to as machine vision or computer vision. Such solutions primarily rely on the application of artificial neural networks. The basic idea underlying these complex algorithms is to mathematically replicate biological neural systems. Frequently, these models are trained using millions of images of well-characterized datasets, such as the ImageNet dataset. In the transfer-learning step, such pre-trained networks are then applied with some modifications to other questions (Table 2).
Recently, AI-assisted approaches have increasingly been evaluated in medical settings and could find more widespread use, especially in imaging disciplines, in the near future. In radiology, for example, a number of companies are already offering products designed to address specific routine diagnostic questions; however, none of these products are being used on a nationwide basis (5).In pathology, this process happens even more hesitantly, because to date whole slide scanners are rarely used for routine diagnosis in Germany. The possible reasons for this are complex, ranging from high costs of investment to security questions to reservations among pathologists. However, some commercial and academic institutions are undertaking pioneering work in this area. In the institutes of pathology in Leeds, Utrecht, Pittsburgh, and New York, digital interpretation has already been partially or predominantly implemented and some companies are already offering specific CE-certified pathology products in certain market segments (5, 6). Besides whole slide scanning, digital reporting comprises automated barcode-based collection of case numbers, speech recognition-assisted dictation of findings and automated transmission of findings to the hospital information system (HIS), among others. A blinded randomized noninferiority study of 1992 cases showed that computer-based digital interpretation was on a par with microscopy-based analog diagnosis (3). In addition, Ho et al. predicted potential cost savings of US$ 12.4 million for a university center over a period of five years, based on model calculations. Of these, US$ 5.4 million are attributable to improved accuracy of diagnoses and resulting reductions in treatment costs (7). Moreover, Stathonikos et al. reported shorter processing times, especially in complex cases (6). However, this exclusively applies to a digital-based approach; for AI applications in pathology, comparable surveys have not yet become available. Both AI support of routine diagnosis in classical pathology and the introduction of novel computer-assisted diagnostic methods are conceivable (Figure 2).
Artificial intelligence to support routine pathological diagnosis
AI and ML algorithms are especially useful for addressing recurrent questions with limited complexity. In routine pathological diagnosis, this includes, among others, analyzes of tissue specimens from large screening programs, such as colorectal cancer screening, and cases where many similar slides are submitted, for example sections of prostate specimens. Shifting time-consuming repetitive screening tasks to AI algorithms would allow pathologists to dedicate more time to more sophisticated activities, such as interpreting predictive or prognostic biomarkers in the context of individual clinical findings. Research based on manual extraction of image properties and classical morphometry dates back to the 1980s (8, 9, 10). At the same time, work to advance the statistical methods was undertaken and first applications for these refined techniques were identified in pathology. In a retrospective analysis, it was possible to differentiate between breast cancer and healthy breast parenchyma with 98.8% accuracy in several benchmark datasets, using feature extraction and a support vector machine (SVM) model-based classifier (11). Similar studies were conducted for prostate cancer and oral cavity squamous cell carcinoma (12, 13). These achieved accuracy levels of 70.8% and 99.1%, respectively. With the help of deep learning and artificial neural networks, classification accuracy could be further improved. In another retrospective study, Ehteshami Bejnordi et al. focused on the detection of lymph node metastases in patients with breast cancer. They compared different models submitted by participants of an annual programming competition in biomedical imaging.The top-performing algorithm, a convolution network based on Google technology, achieved an area under the curve (AUC) value in the receiver operating characteristic curve (AUROC) of 0.994, corresponding to an average false positive rate of 1.25. In the receiver operating characteristic curve, sensitivity is usually plotted against 1-specificity; the integral (AUROC) serves as a quality criterion for predictive power. In addition, the authors compared the best models with the performance of the pathologists. In this comparison, the expert pathologists achieved, in the absence of time constraints, marginally better accuracy values (AUC 0.966) compared to the algorithms (AUC 0.960). However, these values significantly declined as soon as the interpreting pathologist was under time pressure (AUC 0.810) (14). In another study addressing the same question, diagnostic accuracy was assessed with and without AI-based support. In cases with difficult to identify micrometastases, sensitivity improved from 83.3% (human alone) to 91.2% (human in combination with machine) (p = 0.023). In addition, the authors reported higher overall accuracy for the human + machine team compared to interpretation accuracy achieved by human alone or machine alone (15).
The fact that so far only very few pathology laboratories have established a fully digital workflow is a key barrier to the implementation of such models in routine clinical diagnosis. Hybrid methods, including integration of augmented/virtual reality (AR/VR) could be important intermediate steps. Using augmented reality microscopy, where the microscopic image is obtained by a camera and presented to a pre-trained neural network, areas suspicious of containing tumor were marked in the pathologist’s field of view, using a light pointer (16). In Germany, similar approaches are being pursued, for example, in the development of clinical decision support systems (CDSS).
The strategies described to date are primarily based on the supervised learning method. This implies that each image presented to the network during training must first be annotated by a human expert. Since efficient training often requires thousands of example images, this results in a considerable expenditure of work and time. At the same time, expertise in pathology is rare and cannot be easily obtained from other sources, such as crowdsourcing. The method of semi-supervised or unsupervised learning could be a solution to this problem. Multiple instance learning is a method where first all image sections are classified and then only the sections with the lowest classification error are used for training. An annotation of specific image areas is not required; only the entire slide is labeled, e.g. tumor yes/no. Using this method and altogether 44 732 slides of 15 187 patients, Campanella et al. achieved AUROC values of up to 0.991 in the diagnosis of prostate cancer, basal cell carcinoma of the skin and lymph node metastases in a retrospective setting. The discussed implications for routine clinical diagnosis are of interest: If such a system would be used for 65% to 75% of the histopathological specimens received, sensitivity would still be at almost 100% (17).
New diagnostic capabilities through artificial intelligence
Artificial intelligence-assisted image analysis for prognosis prediction
In pathology, important diagnostic queries include estimation of a patient’s prognosis and prediction of potential response to treatment. Frequently, additional immunohistochemical or molecular biological testing, such as next generation sequencing, is used to determine prognostic and predictive biomarkers. Here, again, the application of artificial intelligence in combination with digital pathology would lead to a paradigm shift. AI algorithms can identify subvisual structural characteristics which the human eye is not capable of quantifying, thereby establishing a new class of morphology-based biomarkers with prognostic or predictive validity. In a retrospective study, for example, the prognosis of breast cancer patients was successfully estimated based on hematoxylin-eosin (H&E)–stained slides, using feature extraction and machine learning (18). Image properties were extracted automatically and thousands of properties were identified on cell level and tissue level. The log-rank test found a highly significant association between the prediction of the model and the survival of the patients (p ≤ 0.001), independent of other clinical or pathological risk factors. Patients classified as high-risk patients by the model had a 5-year survival rate of 68.7%. By contrast, 84.5% of the low-risk patients were still alive after five years. In addition, the histopathological criteria contributing most to the respective classification were identified by statistical analyses. A similar method, which was based on deep learning, was described for colorectal cancer (19) and other conditions. First, a deep convolution network was used for image processing. Its output then served as the input of a second network, a so-called recurrent neuronal network with long short-term memory function. Again in retrospective proof-of-concept studies, this method was able to detect microsatellite instability in gastrointestinal tumors and identify the molecular subtype of bladder cancer (20, 21).
Artificial intelligence-based analysis of genetic data
The concept of genotype-phenotype coupling implicates that it is highly likely that in tumor tissue—for example, with a mutation in a key gene—changes in morphology are also found. Besides the ability to predict molecular or clinical parameters based on histomorphological data, AI applications will play an increasingly important role in the analysis of molecular pathological data. One example is the classification of deoxyribonucleic acid (DNA) methylation profiles of squamous cell carcinomas of the lung using deep learning to distinguish a metastasis of a head and neck carcinoma from primary lung cancer (22). Likewise, AI methods play an increasingly important role in multi-omics-based molecular tumor classifications (23). Even though no immediate clinical use has yet been established, evidence of the future complementary significance of complex molecular and morphological profiles is already emerging (24).
Integration of histomorphological, molecular pathological and oncological data
The AI approaches mentioned above allow to predict specific molecular or clinical properties based on histomorphological images or to establish pathological classifications from molecular data. A promising approach to arrive at an even more accurate prognosis or prediction in tumor disease could be to use a combination of various data modalities as input to an AI model. Here, a pathology-specific concept is conceivable that, for example, would integrate histological with immunohistochemical and molecular data. An intriguing example of this was provided by Mobadersany et al. who, in a retrospective study of patients with glioma, included genetic information in addition to image information (25). The combination of image and mutation data achieved better prognostic prediction probability compared to the use of image data alone or mutation data alone (p = 0.0106), expressed as an increase in C index by 5% from 0.754 to 0.801. In the personalized medicine of the future, a multimodal biomarker analysis with integration of morphological, radiographic, laboratory and clinical parameters with genomic and proteomic data could present an enormous challenge due to the complexity of the information that could only be overcome by using AI approaches. However, this would require substantial IT and clinical expertise as well as highest data quality; consequently, the availability of such a solution for routine clinical applications is still a long way off.
Challenges in the implementation of artificial intelligence-based diagnosis
Although artificial intelligence and machine learning have the potential to revolutionize the specialty of pathology, there are a number of significant challenges to their translational implementation. There is a strong positive correlation between the accuracy of an AI algorithm and the amount of data used.In the study by Campanella et al., the validation error decreased approximately by a factor of 10 when 100 times more cases were analyzed (17). However, only a fraction of the histopathological specimens are available in a digital format allowing for computerized analysis. Furthermore, digitalization always requires substantial initial investments. Even though the proportion of digitalized information will significantly increase in the medium term, i.e. in the next decade at the latest, a detailed review and exact description of these data by expert pathologists is lacking. This has major implications for the quality of the data—another factor with significant impact on the accuracy of an AI model (garbage in, garbage out problem). The guide “Digital Pathology“ published by the Professional Association of German Pathologists (Berufsverband Deutscher Pathologen e. V.) offers assistance with the digitalization of pathology. It explicitly encourages the use of digital methods and focusses on the freedom of choice of the method and the pathologist’s responsibility for the selected path to diagnosis (26). At present, the greatest obstacle to the use of AI-based techniques is the almost complete lack of prospective, randomized, multicenter trials evaluating the benefit for pathologists on the one hand and patients on the other hand. Such studies are urgently needed to identify AI solutions that really improve patient-related outcomes. Besides image data, high-quality clinical data are essential for this kind of research. This primarily applies to a potential predictive application of AI-based methods. If these concerns are addressed, the use of AI and digital pathology has the potential to transform the specialty and help pathologists to do their work faster and with greater accuracy.
SF is supported by BMBF research funding (16SV8167), the level I program of the University Medical Center Mainz, by the Mainz Research School of Translational Biomedicine (TransMed), and the Manfred Stolte Foundation.
Conflict of interest statement
Prof. Klauschen is co-founder/shareholder of Aignostics GmbH. He holds a patent with the number US9558550B2. He received consultancy fees from Agilent, BMS, Roche, and Bayer. He received fees for continuing medical education events from BMS, Roche and Bayer.
The remaining authors declare no conflict of interest.
Manuscript received on 24 March 2020; revised version accepted on 10 September 2020
Translated from the original German by Ralf Thoene, MD.
Dr. med. Sebastian Försch
Institut für Pathologie
Langenbeckstr. 1, 55131 Mainz, Germany
Cite this as:
Försch S, Klauschen F, Hufnagl P, Roth W: Artificial intelligence in pathology. Dtsch Arztebl Int 2021; 118: 199–204. DOI: 10.3238/arztebl.m2021.0011
Institute of Pathology, Charité Universitätsmedizin Berlin, Berlin: Prof. Dr. med. Frederick Klauschen, Prof. Dr. rer. nat. Peter Hufnagl
|1.||Schmiegel W, Buchberger B, Follmann M, et al.: S3-Leitlinie – Kolorektales Karzinom. Z Gastroenterol 2017; 55: 1344–498 CrossRef MEDLINE|
|2.||Neumann JH, Jung A, Kirchner T: Molekulare Pathologie des kolorektalen Karzinoms. Pathologe 2015; 36: 137–44 CrossRef MEDLINE|
|3.||Mukhopadhyay S, Feldman MD, Abels E, et al.: Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: a multicenter blinded randomized noninferiority study of 1992 cases (pivotal study). Am J Surg Pathol 2018; 42: 39–52 CrossRef MEDLINE PubMed Central|
|4.||Griffin J, Treanor D: Digital pathology in clinical use: where are we now and what is holding us back? Histopathology 2017; 70: 134–45 CrossRef MEDLINE|
|5.||Bera K, Schalper KA, Rimm DL, Velcheti V, Madabhushi A: Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 2019; 16: 703–15 CrossRef MEDLINE PubMed Central|
|6.||Stathonikos N, Nguyen TQ, Spoto CP, Verdaasdonk MAM, van Diest PJ: Being fully digital: perspective of a Dutch academic pathology laboratory. Histopathology 2019; 75: 621–35 CrossRef MEDLINE PubMed Central|
|7.||Ho J, Ahlers SM, Stratman C, et al.: Can digital pathology result in cost savings? A financial projection for digital pathology implementation at a large integrated health care organization. J Pathol Inform 2014; 5: 33 CrossRef MEDLINE PubMed Central|
|8.||van Der Linden HC, Baak JPA, Lindeman J, Hermans J, Meyer CJ: Morphometry and breast cancer II. Characterisation of breast cancer cells with high malignant potential in patients with spread to lymph nodes: preliminary results. J Clin Pathol 1986; 39: 603–9 CrossRef MEDLINE PubMed Central|
|9.||Baak JPA, Van Der Ley G: Borderline or malignant ovarian tumour? A case report of decision making with morphometry. J Clin Pathol 1984; 37: 1110–3 CrossRef MEDLINE PubMed Central|
|10.||Caspersson TO: Quantitative tumor cytochemistry—G.H.A. Clowes Memorial Lecture. Cancer Res 1979; 39: 2341–5.|
|11.||Osareh A, Shadgar B: A computer aided diagnosis system for breast cancer. IJCSI 2011; 8: 233–40.|
|12.||Lee G, Sparks R, Ali S, et al.: Co-occurring gland angularity in localized subgraphs: predicting biochemical recurrence in intermediate-risk prostate cancer patients. PLoS One 2014; 9: e97954 CrossRef MEDLINE PubMed Central|
|13.||Lu C, Lewis JS Jr, Dupont WD, Plummer WD Jr, Janowczyk A, Madabhushi A: An oral cavity squamous cell carcinoma quantitative histomorphometric-based image classifier of nuclear morphology can risk stratify patients for disease-specific survival. Mod Pathol 2017; 30: 1655–65 CrossRef MEDLINE PubMed Central|
|14.||Ehteshami Bejnordi B, Veta M, Johannes van Diest P, et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 2017; 318: 2199–2210 CrossRef MEDLINE PubMed Central|
|15.||Steiner DF, Macdonald R, Liu Y, et al.: Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol 2018; 42: 1636–46 CrossRef MEDLINE PubMed Central|
|16.||Chen PC, Gadepalli K, MacDonald R, et al.: An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis. Nat Med 2019; 25: 1453–7 CrossRef MEDLINE|
|17.||Campanella G, Hanna MG, Geneslaw L, et al.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 2019; 25: 1301–9 CrossRef MEDLINE PubMed Central|
|18.||Beck AH, Sangoi AR, Leung S, et al.: Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med 2011; 3: 108ra113 CrossRef MEDLINE|
|19.||Bychkov D, Linder N, Turkki R, et al.: Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci Rep 2018; 8: 3395 CrossRef MEDLINE PubMed Central|
|20.||Kather JN, Pearson AT, Halama N, et al.: Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 2019; 25: 1054–6 CrossRef MEDLINE PubMed Central|
|21.||Woerl AC, Eckstein M, Geiger J, et al.: Deep learning predicts molecular subtype of muscle-invasive bladder cancer from conventional histopathological slides. Eur Urol 2020; 78: 256–64 CrossRef MEDLINE|
|22.||Jurmeister P, Bockmayr M, Seegerer P, et al.: Machine learning analysis of DNA methylation profiles distinguishes primary lung squamous cell carcinomas from head and neck metastases. Sci Transl Med 2019; 11: eaaw8513 CrossRef MEDLINE|
|23.||Hoadley KA, Yau C, Hinoue T, et al.: Cell-of-origin patterns dominate the molecular classification of 10,000 tumors from 33 types of cancer. Cell 2018; 173: 291–304 CrossRef MEDLINE PubMed Central|
|24.||Hoberger M, von Laffert M, Heim D, Klauschen F: Histomorphological and molecular profiling: friends not foes! Morpho-molecular analysis reveals agreement between histological and molecular profiling. Histopathology 2019; 75: 694–703 CrossRef MEDLINE|
|25.||Mobadersany P, Yousefi S, Amgad M, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci USA 2018; 115: E2970-9 CrossRef MEDLINE PubMed Central|
|26.||Haroske G, Zwönitzer R, Hufnagl P, Kommission Digitale Pathologie: Leitfaden „Digitale Pathologie in der Diagnostik“: Befunderstellung an digitalen Bildern. Pathologe 2018; 39: 216–21 CrossRef MEDLINE|
|27.||Konrad E: Zur Geschichte der künstlichen Intelligenz in der Bundesrepublik Deutschland. In: Siefkes D, Eulenhöfer P, Stach H, Städtler K (eds): Sozialgeschichte der Informatik - Kulturelle Praktiken und Orientierungen. Wiesbaden: Deutscher Universitäts-Verlag, Springer Fachmedien Wiesbaden GmbH 1998: 287–96 CrossRef|
|28.||Dechter R: Learning while searching in constraint-satisfaction-problems. AAAI-86 Proceedings 1986; 178–83.|
|29.||LeCun Y, Boser B, Denker JS, et al.: Backpropagation applied to digit recognition. Neural computation 1989; 541–51 CrossRef|
|30.||Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L: ImageNet: a large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Miami, FL 2009; 248–55 CrossRef|
|31.||Goodfellow IJ, Pouget-Abadie J, Mirza M, et al.: Generative adversarial networks. Adv Neural Inf Process Syst 2014; 3: 2672–80.|