Predicting cell types from H&E using multiplex imaging and deep learning

Session

Poster Session

Authors

Rolf Harkes (1), Ynze van Cleef (1), Idris Iritas (1), Hugo Horlings (1)

Affiliations

1. Netherlands Cancer Institute

Keywords

Pathology, H&E, CODEX, multiplex imaging, deep learning, AI

Abstract text

Using the process of pattern recognition, a pathologist examines a tissue slide and looks for criteria of normality and abnormality within the tissue. Pattern recognition in pathology depends on linking the observed patterns of individual cell types in human body organs with criteria representing specific diseases. In clinical practice a pathologist diagnoses resected cancer samples. The tissue samples are typically stained with Haematoxylin and Eosin (H&E) that mark the nucleus and the extracellular matrix, respectively of each cell. Pathologists use patterns, quantity and location of tumour cells, lymphocytes and fibroblasts. With these parameters they determine a diagnosis and the characteristics that are associated with survival outcome of the patient (prognosis) and the treatment response. However, the assessment of the large amount of cells by pathologists is time-consuming, expensive and known to be subject to error [1].

The development of slide scanners and the resulting digitization has enabled machine learning to assist pathologists during this process. Deep learning models are already in use for many segmentation and classification tasks in image analysis. A specific model that could identify tumour cells, lymphocytes and fibroblasts in triple negative breast cancer (TNBC) would be of great use for predicting immunotherapy response [2][3]. However, to train these models requires a large volume of trustworthy ground truth labels.

Here we present the use of multiplex imaging to accurately identify different cell types in tissue micro arrays (TMAs) as well as whole slide images (WSI) of human TNBC tissue. The CODEX fusion technology[4] is used to stain and image 36 different markers. This gives a very detailed insight into the spatial distribution of different cell types. The slides are subsequently stained with H&E and the same image is registered to the multiplex image to correctly localise the classified cells in the H&E. We show that our attention U-NET model is able to accurately predict tumour cells, fibroblasts and lymphocytes from H&E images with an overall accuracy of 84%. We further analyse the ability of our model to predict different lymphocyte subtypes. We plan to look at the morphological features in the H&E that are responsible for the prediction of our model.

The analysis pipeline that we have developed is free and open source. It is built with QuPath[5], ImageJ[6] and Python and is available on GitHub.

References

[1] Peck, M., Moffat, D., Latham, B., and Badrick, T. (2018). Review of diagnostic error in anatomical pathology and the role and value of second opinions in error prevention. Journal of clinical pathology, 71(11):995–1000.

[2] Voorwerk, L., Slagter, M., Horlings, H.M. et al. Immune induction strategies in metastatic triple-negative breast cancer to enhance the sensitivity to PD-1 blockade: the TONIC trial. Nat Med 25, 920–928 (2019)

[3] Nederlof, I., Hajizadeh, S., Sobhani, F. et al. Spatial interplay of lymphocytes and fibroblasts in estrogen receptor-positive HER2-negative breast cancer. npj Breast Cancer 8, 56 (2022)

[4] Black, S., Phillips, D., Hickey, J.W. et al. CODEX multiplexed tissue imaging with DNA-conjugated antibodies. Nat Protoc 16, 3802–3835 (2021)

[5] Bankhead, P., Loughrey, M. B., Fern´andez, J. A., Dombrowski, Y., McArt, D. G., Dunne, P. D., McQuaid, S., Gray, R. T., Murray, L. J., Coleman, H. G., et al. (2017). Qupath: Open source software for digital pathology image analysis. Scientific reports, 7(1):1–7.

[6] Schindelin, J., Arganda-Carreras, I., Frise, E. et al. Fiji: an open-source platform for biological-image analysis.  Nat Methods 9, 676–682 (2012).