Generalising multistain immunohistochemistry tissue segmentation using end-to-end colour deconvolution deep neural networks

Generalising multistain immunohistochemistry tissue segmentation using end-to-end colour deconvolution deep neural networks

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for $120.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Image Processing — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

A key challenge in cancer immunotherapy biomarker research is quantification of pattern changes in microscopic whole slide images of tumour biopsies. Drug development requires a correlative analysis of various biomarkers. To enable that, tissue slides are manually annotated by pathologists, which is a tedious and error-prone task. Automation of this annotation process can improve accuracy and consistency while reducing workload and cost. The authors present a deep learning method to automatically segment digitised slide images with multiple stainings into compartments of tumour, healthy tissue, necrosis, and background. The method is based on using a fully convolutional neural network including a colour deconvolution segment learned end-to-end and helping the network to converge faster and deal with the dataset staining variability. They evaluate the performance of the proposed method using the F1 score, which is the harmonic mean between precision and recall. They report a testing F1 score of 0.88, 0.9, 0.8, and 0.99 for tumour, tissue, necrosis, and background, respectively. They address the task in the context of drug development where multiple stains exist and look into solutions for generalisations over these image populations. They also apply visualisation techniques to help understand the network decisions and gain more trust from pathologists.


    1. 1)
      • 1. Zhang, W., Li, R., Deng, H., et al: ‘Deep convolutional neural networks for multi-modality isointense infant brain image segmentation’, NeuroImage, 2015, 108, pp. 214224.
    2. 2)
      • 2. Birenbaum, A., Greenspan, H.: ‘Longitudinal multiple sclerosis lesion segmentation using multi-view convolutional neural networks’. Deep Learning Data Labeling Medical Applications, Athens, Greece, 2016, pp. 5867.
    3. 3)
      • 3. Brosch, T., Tang, L.Y., Yoo, Y., et al: ‘Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation’, Trans. Med. Imaging, 2016, 35, (5), pp. 12291239.
    4. 4)
      • 4. Havaei, M., Guizard, N., Chapados, N., et al: ‘HeMIS: hetero-modal image segmentation’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 469477.
    5. 5)
      • 5. Kamnitsas, K., Ledig, C., Newcombe, V.F., et al: ‘Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation’, Med. Image Anal., 2017, 36, pp. 6178.
    6. 6)
      • 6. Pereira, S., Pinto, A., Alves, V., et al: ‘Brain tumor segmentation using convolutional neural networks in MRI images’, Trans. Med. Imaging, 2016, 35, (5), pp. 12401251.
    7. 7)
      • 7. Zhao, L., Jia, K.: ‘Multiscale CNNs for brain tumor segmentation and diagnosis’, Comput. Math. Methods Med., 2016, 2016, pp. 17.
    8. 8)
      • 8. Stollenga, M.F., Byeon, W., Liwicki, M., et al: ‘Parallel multidimensional LSTM, with application to fast biomedical volumetric image segmentation’. Neural Information Processing Systems (NIPS), Montreal, Canada, 2015, pp. 29983006.
    9. 9)
      • 9. Ciresan, D., Giusti, A., Gambardella, L.M., et al: ‘Deep neural networks segment neuronal membranes in electron microscopy images’. Neural Information Processing Systems (NIPS), Lake Tahoe, USA, 2012, pp. 28432851.
    10. 10)
      • 10. Ronneberger, O., Fischer, P., Brox, T.: ‘U-net: convolutional networks for biomedical image segmentation’. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, 2015, pp. 234241.
    11. 11)
      • 11. Roth, H.R., Farag, A., Lu, L., et al: ‘Deep convolutional networks for pancreas segmentation in CT imaging’. SPIE Medical Imaging 2015: Image Processing, San Diego, USA, 2015, vol. 9413, p. 94131G.
    12. 12)
      • 12. Prasoon, A., Petersen, K., Igel, C., et al: ‘Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Nagoya, Japan, 2013, pp. 246253.
    13. 13)
      • 13. Wang, D., Khosla, A., Gargeya, R., et al: ‘Deep learning for identifying metastatic breast cancer’, arXiv preprint arXiv:160605718, 2016.
    14. 14)
      • 14. Liu, Y., Gadepalli, K., Norouzi, M., et al: ‘Detecting cancer metastases on gigapixel pathology images’, arXiv preprint arXiv:170302442, 2017.
    15. 15)
      • 15. Veta, M., Pluim, J.P., Van Diest, P.J., et al: ‘Corrections to ‘breast cancer histopathology image analysis: a review’ [May 14 1400-1411]’, IEEE Trans. Biomed. Eng., 2014, 61, (11), pp. 28192819.
    16. 16)
      • 16. Bhargava, R., Madabhushi, A.: ‘Emerging themes in image informatics and molecular analysis for digital pathology’, Annu. Rev. Biomed. Eng., 2016, 18, pp. 387412.
    17. 17)
      • 17. Meyer, J.S., Alvarez, C., Milikowski, C., et al: ‘Breast carcinoma malignancy grading by bloom–Richardson system vs proliferation index: reproducibility of grade and advantages of proliferation index’, Mod. Pathol., 2005, 18, (8), p. 1067.
    18. 18)
      • 18. Song, Y., Tan, E.L., Jiang, X., et al: ‘Accurate cervical cell segmentation from overlapping clumps in pap smear images’, Trans. Med. Imaging, 2017, 36, (1), pp. 288300.
    19. 19)
      • 19. Xing, F., Xie, Y., Yang, L.: ‘An automatic learning-based framework for robust nucleus segmentation’, Trans. Med. Imaging, 2016, 35, (2), pp. 550566.
    20. 20)
      • 20. Janowczyk, A., Doyle, S., Gilmore, H., et al: ‘A resolution adaptive deep hierarchical (RADHical) learning scheme applied to nuclear segmentation of digital pathology images’, Comput. Methods Biomech. Biomed. Eng. Imaging Vis., 2018, 6, (3), pp. 270276.
    21. 21)
      • 21. BenTaieb, A., Hamarneh, G.: ‘Topology aware fully convolutional networks for histology gland segmentation’. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 460468.
    22. 22)
      • 22. BenTaieb, A., Kawahara, J., Hamarneh, G.: ‘Multi-loss convolutional networks for gland analysis in microscopy’. IEEE 13th Int. Symp. on Biomedical Imaging (ISBI), Prague, Czech Republic, 2016, pp. 642645.
    23. 23)
      • 23. Chen, H., Qi, X., Yu, L., et al: ‘DCAN: deep contour-aware networks for object instance segmentation from histology images’, Med. Image Anal., 2017, 36, pp. 135146.
    24. 24)
      • 24. Li, W., Manivannan, S., Akbar, S., et al: ‘Gland segmentation in colon histology images using hand-crafted features and convolutional neural networks’. IEEE 13th Int. Symp. on Biomedical Imaging (ISBI), Prague, Czech Republic, 2016, pp. 14051408.
    25. 25)
      • 25. Xu, Y., Li, Y., Liu, M., et al: ‘Gland instance segmentation by deep multichannel side supervision’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 496504.
    26. 26)
      • 26. Sirinukunwattana, K., Pluim, J.P., Chen, H., et al: ‘Gland segmentation in colon histology images: the glas challenge contest’, Med. Image Anal., 2017, 35, pp. 489502.
    27. 27)
      • 27. Kainz, P., Pfeiffer, M., Urschler, M.: ‘Semantic segmentation of colon glands with deep convolutional neural networks and total variation segmentation’, arXiv preprint arXiv:151106919, 2015.
    28. 28)
      • 28. Xu, J., Luo, X., Wang, G., et al: ‘A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images’, Neurocomputing, 2016, 191, pp. 214223.
    29. 29)
      • 29. Xie, Y., Zhang, Z., Sapkota, M., et al: ‘Spatial clockwork recurrent neural network for muscle perimysium segmentation’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 185193.
    30. 30)
      • 30. Wang, J., MacKenzie, J.D., Ramachandran, R., et al: ‘A deep learning approach for semantic segmentation in histology tissue images’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 176184.
    31. 31)
      • 31. Apou, G., Schaadt, N.S., Naegel, B., et al: ‘Detection of lobular structures in normal breast tissue’, Comput. Biol. Med., 2016, 74, pp. 91102.
    32. 32)
      • 32. Cruz-Roa, A., Basavanhally, A., González, F., et al: ‘Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks’. SPIE Medical Imaging 2014: Digital Pathology, San Diego, USA, 2014, vol. 9041, p. 904103.
    33. 33)
      • 33. Litjens, G., Sánchez, C.I., Timofeeva, N., et al: ‘Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis’, Sci. Rep., 2016, 6, p. 26286.
    34. 34)
      • 34. Rezaeilouyeh, H., Mollahosseini, A., Mahoor, M.H.: ‘Microscopic medical image classification framework via deep learning and shearlet transform’, J. Med. Imaging, 2016, 3, (4), p. 044501.
    35. 35)
      • 35. Eigen, D., Fergus, R.: ‘Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture’. Int. Conf. on Computer Vision (ICCV), Las Condes, Chile, 2015, pp. 26502658.
    36. 36)
      • 36. Bottou, L.: ‘Stochastic gradient learning in neural networks’, Proc. Neuro-Nımes, 1991, 91, (8), p. 12.
    37. 37)
      • 37. Das, D., Avancha, S., Mudigere, D., et al: ‘Distributed deep learning using synchronous stochastic gradient descent’, arXiv preprint arXiv:160206709, 2016.
    38. 38)
      • 38. Paszke, A., Gross, S., Chintala, S., et al: ‘Pytorch’, 2017.
    39. 39)
      • 39. Chinchor, N.: ‘Muc-4 evaluation metrics’. Proc. 4th Conf. on Message Understanding, Virginia, USA, 1992, pp. 2229.
    40. 40)
      • 40. Ioffe, S., Szegedy, C.: ‘Batch normalization: accelerating deep network training by reducing internal covariate shift’, arXiv preprint arXiv:150203167, 2015.
    41. 41)
      • 41. Srivastava, N., Hinton, G., Krizhevsky, A., et al: ‘Dropout: a simple way to prevent neural networks from overfitting’, J. Mach. Learn. Res., 2014, 15, (1), pp. 19291958.
    42. 42)
      • 42. Sirinukunwattana, K., Raza, S.E.A., Tsang, Y.W., et al: ‘Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images’, Trans. Med. Imaging, 2016, 35, (5), pp. 11961206.
    43. 43)
      • 43. Youssif, A.A., Gawish, A.S., Moussa, M.E.: ‘Automated periodontal diseases classification system’, Editorial Preface, 2012, 3, (1), pp. 4048.
    44. 44)
      • 44. Chen, T., Chefd'Hotel, C.: ‘Deep learning based automatic immune cell detection for immunohistochemistry images’. Machine Learning in Medical Imaging (MLMI), Boston, USA, 2014, pp. 1724.
    45. 45)
      • 45. Duggal, R., Gupta, A., Gupta, R., et al: ‘SD-layer: stain deconvolutional layer for CNNs in medical microscopic imaging’. Medical Image Computing and Computer Assisted Intervention (MICCAI), Quebec City, Canada, 2017, pp. 435443.
    46. 46)
      • 46. Li, X., Plataniotis, K.N.: ‘A complete color normalization approach to histopathology images using color cues computed from saturation-weighted statistics’, IEEE Trans. Biomed. Eng., 2015, 62, (7), pp. 18621873.
    47. 47)
      • 47. Ruifrok, A.C., Johnston, D.A.: ‘Quantification of histochemical staining by color deconvolution’, Anal. Quant. Cytol. Histol., 2001, 23, (4), pp. 291299.
    48. 48)
      • 48. Xu, J., Xiang, L., Wang, G., et al: ‘Sparse non-negative matrix factorization (SNMF) based color unmixing for breast histopathological image analysis’, Comput. Med. Imaging Graph., 2015, 46, pp. 2029.
    49. 49)
      • 49. Badrinarayanan, V., Kendall, A., Cipolla, R.: ‘Segnet: a deep convolutional encoder-decoder architecture for image segmentation’, arXiv preprint arXiv:151100561, 2015.
    50. 50)
      • 50. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016, pp. 770778.
    51. 51)
      • 51. Olah, C., Mordvintsev, A., Schubert, L.: ‘Feature visualization’, Distill, 2017, 2, (11), pp. e7e7. Available at
    52. 52)
      • 52. Yosinski, J., Clune, J., Nguyen, A., et al: ‘Understanding neural networks through deep visualization’, arXiv preprint arXiv:150606579, 2015.
    53. 53)
      • 53. Nguyen, A., Yosinski, J., Clune, J.: ‘Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks’, arXiv preprint arXiv:160203616, 2016.
    54. 54)
      • 54. Nguyen, A., Dosovitskiy, A., Yosinski, J., et al: ‘Synthesizing the preferred inputs for neurons in neural networks via deep generator networks’. Neural Information Processing Systems (NIPS), Barcelona, Spain, 2016, pp. 33873395.
    55. 55)
      • 55. Zeiler, M.D., Fergus, R.: ‘Visualizing and understanding convolutional networks’. European Conf. on Computer Vision (ECCV), Zurich, Switzerland, 2014, pp. 818833.
    56. 56)
      • 56. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ‘Imagenet classification with deep convolutional neural networks’. Neural Information Processing Systems (NIPS), Lake Tahoe, USA, 2012, pp. 10971105.
    57. 57)
      • 57. Mahendran, A., Vedaldi, A.: ‘Visualizing deep convolutional neural networks using natural pre-images’, Int. J. Comput. Vis., 2016, 120, (3), pp. 233255.
    58. 58)
      • 58. Simonyan, K., Vedaldi, A., Zisserman, A.: ‘Deep inside convolutional networks: visualising image classification models and saliency maps’, arXiv preprint arXiv:13126034, 2013.
    59. 59)
      • 59. Smilkov, D., Thorat, N., Kim, B., et al: ‘Smoothgrad: removing noise by adding noise’, arXiv preprint arXiv:170603825, 2017.
    60. 60)
      • 60. Albarqouni, S., Baur, C., Achilles, F., et al: ‘Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images’, Trans. Med. Imaging, 2016, 35, (5), pp. 13131321.
    61. 61)
      • 61. Khan, A.M., Rajpoot, N., Treanor, D., et al: ‘A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution’, IEEE Trans. Biomed. Eng., 2014, 61, (6), pp. 17291738.
    62. 62)
      • 62. Khan, A.M., El-Daly, H., Simmons, E., et al: ‘Hymap: a hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images’, J. Pathol. Inform., 2013, 4, (Suppl), pp. s1s1.
    63. 63)
      • 63. Alsubaie, N., Trahearn, N., Raza, S.E.A., et al: ‘Stain deconvolution using statistical analysis of multi-resolution stain colour representation’, PloS one, 2017, 12, (1), p. e0169875.

Related content

This is a required field
Please enter a valid email address