access icon free Generalising multistain immunohistochemistry tissue segmentation using end-to-end colour deconvolution deep neural networks

© The Institution of Engineering and Technology
Received 20/11/2018, Accepted 11/02/2019, Revised 24/01/2019, Published 26/02/2019

A key challenge in cancer immunotherapy biomarker research is quantification of pattern changes in microscopic whole slide images of tumour biopsies. Drug development requires a correlative analysis of various biomarkers. To enable that, tissue slides are manually annotated by pathologists, which is a tedious and error-prone task. Automation of this annotation process can improve accuracy and consistency while reducing workload and cost. The authors present a deep learning method to automatically segment digitised slide images with multiple stainings into compartments of tumour, healthy tissue, necrosis, and background. The method is based on using a fully convolutional neural network including a colour deconvolution segment learned end-to-end and helping the network to converge faster and deal with the dataset staining variability. They evaluate the performance of the proposed method using the F1 score, which is the harmonic mean between precision and recall. They report a testing F1 score of 0.88, 0.9, 0.8, and 0.99 for tumour, tissue, necrosis, and background, respectively. They address the task in the context of drug development where multiple stains exist and look into solutions for generalisations over these image populations. They also apply visualisation techniques to help understand the network decisions and gain more trust from pathologists.

Inspec keywords: cancer; biomedical optical imaging; learning (artificial intelligence); drugs; convolutional neural nets; tumours; image segmentation; medical image processing

Other keywords: multistain immunohistochemistry tissue segmentation; tumour biopsies; colour deconvolution segment; biomarkers; microscopic whole slide images; colour deconvolution deep neural networks; image populations; cancer immunotherapy biomarker research; fully convolutional neural network; annotation process; deep learning method; drug development; segment digitised slide images

Subjects: Optical and laser radiation (biomedical imaging/measurement); Optical and laser radiation (medical uses); Computer vision and image processing techniques; Neural computing techniques; Patient diagnostic methods and instrumentation; Optical, image and video signal processing; Biology and medical computing

References

    1. 1)
      • 57. Mahendran, A., Vedaldi, A.: ‘Visualizing deep convolutional neural networks using natural pre-images’, Int. J. Comput. Vis., 2016, 120, (3), pp. 233–255.
    2. 2)
      • 32. Cruz-Roa, A., Basavanhally, A., González, F., et al: ‘Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks’. SPIE Medical Imaging 2014: Digital Pathology, San Diego, USA, 2014, vol. 9041, p. 904103.
    3. 3)
      • 40. Ioffe, S., Szegedy, C.: ‘Batch normalization: accelerating deep network training by reducing internal covariate shift’, arXiv preprint arXiv:150203167, 2015.
    4. 4)
      • 8. Stollenga, M.F., Byeon, W., Liwicki, M., et al: ‘Parallel multidimensional LSTM, with application to fast biomedical volumetric image segmentation’. Neural Information Processing Systems (NIPS), Montreal, Canada, 2015, pp. 2998–3006.
    5. 5)
      • 11. Roth, H.R., Farag, A., Lu, L., et al: ‘Deep convolutional networks for pancreas segmentation in CT imaging’. SPIE Medical Imaging 2015: Image Processing, San Diego, USA, 2015, vol. 9413, p. 94131G.
    6. 6)
      • 29. Xie, Y., Zhang, Z., Sapkota, M., et al: ‘Spatial clockwork recurrent neural network for muscle perimysium segmentation’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 185–193.
    7. 7)
      • 56. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ‘Imagenet classification with deep convolutional neural networks’. Neural Information Processing Systems (NIPS), Lake Tahoe, USA, 2012, pp. 1097–1105.
    8. 8)
      • 45. Duggal, R., Gupta, A., Gupta, R., et al: ‘SD-layer: stain deconvolutional layer for CNNs in medical microscopic imaging’. Medical Image Computing and Computer Assisted Intervention (MICCAI), Quebec City, Canada, 2017, pp. 435–443.
    9. 9)
      • 1. Zhang, W., Li, R., Deng, H., et al: ‘Deep convolutional neural networks for multi-modality isointense infant brain image segmentation’, NeuroImage, 2015, 108, pp. 214–224.
    10. 10)
      • 63. Alsubaie, N., Trahearn, N., Raza, S.E.A., et al: ‘Stain deconvolution using statistical analysis of multi-resolution stain colour representation’, PloS one, 2017, 12, (1), p. e0169875.
    11. 11)
      • 17. Meyer, J.S., Alvarez, C., Milikowski, C., et al: ‘Breast carcinoma malignancy grading by bloom–Richardson system vs proliferation index: reproducibility of grade and advantages of proliferation index’, Mod. Pathol., 2005, 18, (8), p. 1067.
    12. 12)
      • 33. Litjens, G., Sánchez, C.I., Timofeeva, N., et al: ‘Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis’, Sci. Rep., 2016, 6, p. 26286.
    13. 13)
      • 46. Li, X., Plataniotis, K.N.: ‘A complete color normalization approach to histopathology images using color cues computed from saturation-weighted statistics’, IEEE Trans. Biomed. Eng., 2015, 62, (7), pp. 1862–1873.
    14. 14)
      • 60. Albarqouni, S., Baur, C., Achilles, F., et al: ‘Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images’, Trans. Med. Imaging, 2016, 35, (5), pp. 1313–1321.
    15. 15)
      • 7. Zhao, L., Jia, K.: ‘Multiscale CNNs for brain tumor segmentation and diagnosis’, Comput. Math. Methods Med., 2016, 2016, pp. 1–7.
    16. 16)
      • 61. Khan, A.M., Rajpoot, N., Treanor, D., et al: ‘A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution’, IEEE Trans. Biomed. Eng., 2014, 61, (6), pp. 1729–1738.
    17. 17)
      • 28. Xu, J., Luo, X., Wang, G., et al: ‘A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images’, Neurocomputing, 2016, 191, pp. 214–223.
    18. 18)
      • 26. Sirinukunwattana, K., Pluim, J.P., Chen, H., et al: ‘Gland segmentation in colon histology images: the glas challenge contest’, Med. Image Anal., 2017, 35, pp. 489–502.
    19. 19)
      • 58. Simonyan, K., Vedaldi, A., Zisserman, A.: ‘Deep inside convolutional networks: visualising image classification models and saliency maps’, arXiv preprint arXiv:13126034, 2013.
    20. 20)
      • 34. Rezaeilouyeh, H., Mollahosseini, A., Mahoor, M.H.: ‘Microscopic medical image classification framework via deep learning and shearlet transform’, J. Med. Imaging, 2016, 3, (4), p. 044501.
    21. 21)
      • 59. Smilkov, D., Thorat, N., Kim, B., et al: ‘Smoothgrad: removing noise by adding noise’, arXiv preprint arXiv:170603825, 2017.
    22. 22)
      • 36. Bottou, L.: ‘Stochastic gradient learning in neural networks’, Proc. Neuro-Nımes, 1991, 91, (8), p. 12.
    23. 23)
      • 31. Apou, G., Schaadt, N.S., Naegel, B., et al: ‘Detection of lobular structures in normal breast tissue’, Comput. Biol. Med., 2016, 74, pp. 91–102.
    24. 24)
      • 3. Brosch, T., Tang, L.Y., Yoo, Y., et al: ‘Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation’, Trans. Med. Imaging, 2016, 35, (5), pp. 1229–1239.
    25. 25)
      • 39. Chinchor, N.: ‘Muc-4 evaluation metrics’. Proc. 4th Conf. on Message Understanding, Virginia, USA, 1992, pp. 22–29.
    26. 26)
      • 9. Ciresan, D., Giusti, A., Gambardella, L.M., et al: ‘Deep neural networks segment neuronal membranes in electron microscopy images’. Neural Information Processing Systems (NIPS), Lake Tahoe, USA, 2012, pp. 2843–2851.
    27. 27)
      • 22. BenTaieb, A., Kawahara, J., Hamarneh, G.: ‘Multi-loss convolutional networks for gland analysis in microscopy’. IEEE 13th Int. Symp. on Biomedical Imaging (ISBI), Prague, Czech Republic, 2016, pp. 642–645.
    28. 28)
      • 30. Wang, J., MacKenzie, J.D., Ramachandran, R., et al: ‘A deep learning approach for semantic segmentation in histology tissue images’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 176–184.
    29. 29)
      • 38. Paszke, A., Gross, S., Chintala, S., et al: ‘Pytorch’, 2017.
    30. 30)
      • 53. Nguyen, A., Yosinski, J., Clune, J.: ‘Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks’, arXiv preprint arXiv:160203616, 2016.
    31. 31)
      • 54. Nguyen, A., Dosovitskiy, A., Yosinski, J., et al: ‘Synthesizing the preferred inputs for neurons in neural networks via deep generator networks’. Neural Information Processing Systems (NIPS), Barcelona, Spain, 2016, pp. 3387–3395.
    32. 32)
      • 21. BenTaieb, A., Hamarneh, G.: ‘Topology aware fully convolutional networks for histology gland segmentation’. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 460–468.
    33. 33)
      • 27. Kainz, P., Pfeiffer, M., Urschler, M.: ‘Semantic segmentation of colon glands with deep convolutional neural networks and total variation segmentation’, arXiv preprint arXiv:151106919, 2015.
    34. 34)
      • 15. Veta, M., Pluim, J.P., Van Diest, P.J., et al: ‘Corrections to ‘breast cancer histopathology image analysis: a review’ [May 14 1400-1411]’, IEEE Trans. Biomed. Eng., 2014, 61, (11), pp. 2819–2819.
    35. 35)
      • 25. Xu, Y., Li, Y., Liu, M., et al: ‘Gland instance segmentation by deep multichannel side supervision’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 496–504.
    36. 36)
      • 51. Olah, C., Mordvintsev, A., Schubert, L.: ‘Feature visualization’, Distill, 2017, 2, (11), pp. e7–e7. Available at https://doi.org/10.23915%2Fdistill.00007.
    37. 37)
      • 2. Birenbaum, A., Greenspan, H.: ‘Longitudinal multiple sclerosis lesion segmentation using multi-view convolutional neural networks’. Deep Learning Data Labeling Medical Applications, Athens, Greece, 2016, pp. 58–67.
    38. 38)
      • 13. Wang, D., Khosla, A., Gargeya, R., et al: ‘Deep learning for identifying metastatic breast cancer’, arXiv preprint arXiv:160605718, 2016.
    39. 39)
      • 47. Ruifrok, A.C., Johnston, D.A.: ‘Quantification of histochemical staining by color deconvolution’, Anal. Quant. Cytol. Histol., 2001, 23, (4), pp. 291–299.
    40. 40)
      • 10. Ronneberger, O., Fischer, P., Brox, T.: ‘U-net: convolutional networks for biomedical image segmentation’. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, 2015, pp. 234–241.
    41. 41)
      • 23. Chen, H., Qi, X., Yu, L., et al: ‘DCAN: deep contour-aware networks for object instance segmentation from histology images’, Med. Image Anal., 2017, 36, pp. 135–146.
    42. 42)
      • 18. Song, Y., Tan, E.L., Jiang, X., et al: ‘Accurate cervical cell segmentation from overlapping clumps in pap smear images’, Trans. Med. Imaging, 2017, 36, (1), pp. 288–300.
    43. 43)
      • 48. Xu, J., Xiang, L., Wang, G., et al: ‘Sparse non-negative matrix factorization (SNMF) based color unmixing for breast histopathological image analysis’, Comput. Med. Imaging Graph., 2015, 46, pp. 20–29.
    44. 44)
      • 52. Yosinski, J., Clune, J., Nguyen, A., et al: ‘Understanding neural networks through deep visualization’, arXiv preprint arXiv:150606579, 2015.
    45. 45)
      • 62. Khan, A.M., El-Daly, H., Simmons, E., et al: ‘Hymap: a hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images’, J. Pathol. Inform., 2013, 4, (Suppl), pp. s1–s1.
    46. 46)
      • 14. Liu, Y., Gadepalli, K., Norouzi, M., et al: ‘Detecting cancer metastases on gigapixel pathology images’, arXiv preprint arXiv:170302442, 2017.
    47. 47)
      • 19. Xing, F., Xie, Y., Yang, L.: ‘An automatic learning-based framework for robust nucleus segmentation’, Trans. Med. Imaging, 2016, 35, (2), pp. 550–566.
    48. 48)
      • 37. Das, D., Avancha, S., Mudigere, D., et al: ‘Distributed deep learning using synchronous stochastic gradient descent’, arXiv preprint arXiv:160206709, 2016.
    49. 49)
      • 49. Badrinarayanan, V., Kendall, A., Cipolla, R.: ‘Segnet: a deep convolutional encoder-decoder architecture for image segmentation’, arXiv preprint arXiv:151100561, 2015.
    50. 50)
      • 55. Zeiler, M.D., Fergus, R.: ‘Visualizing and understanding convolutional networks’. European Conf. on Computer Vision (ECCV), Zurich, Switzerland, 2014, pp. 818–833.
    51. 51)
      • 41. Srivastava, N., Hinton, G., Krizhevsky, A., et al: ‘Dropout: a simple way to prevent neural networks from overfitting’, J. Mach. Learn. Res., 2014, 15, (1), pp. 1929–1958.
    52. 52)
      • 35. Eigen, D., Fergus, R.: ‘Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture’. Int. Conf. on Computer Vision (ICCV), Las Condes, Chile, 2015, pp. 2650–2658.
    53. 53)
      • 4. Havaei, M., Guizard, N., Chapados, N., et al: ‘HeMIS: hetero-modal image segmentation’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Athens, Greece, 2016, pp. 469–477.
    54. 54)
      • 16. Bhargava, R., Madabhushi, A.: ‘Emerging themes in image informatics and molecular analysis for digital pathology’, Annu. Rev. Biomed. Eng., 2016, 18, pp. 387–412.
    55. 55)
      • 43. Youssif, A.A., Gawish, A.S., Moussa, M.E.: ‘Automated periodontal diseases classification system’, Editorial Preface, 2012, 3, (1), pp. 40–48.
    56. 56)
      • 42. Sirinukunwattana, K., Raza, S.E.A., Tsang, Y.W., et al: ‘Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images’, Trans. Med. Imaging, 2016, 35, (5), pp. 1196–1206.
    57. 57)
      • 50. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016, pp. 770–778.
    58. 58)
      • 44. Chen, T., Chefd'Hotel, C.: ‘Deep learning based automatic immune cell detection for immunohistochemistry images’. Machine Learning in Medical Imaging (MLMI), Boston, USA, 2014, pp. 17–24.
    59. 59)
      • 6. Pereira, S., Pinto, A., Alves, V., et al: ‘Brain tumor segmentation using convolutional neural networks in MRI images’, Trans. Med. Imaging, 2016, 35, (5), pp. 1240–1251.
    60. 60)
      • 5. Kamnitsas, K., Ledig, C., Newcombe, V.F., et al: ‘Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation’, Med. Image Anal., 2017, 36, pp. 61–78.
    61. 61)
      • 12. Prasoon, A., Petersen, K., Igel, C., et al: ‘Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network’. Medical Image Computing and Computer-Assisted Intervention (MICCAI), Nagoya, Japan, 2013, pp. 246–253.
    62. 62)
      • 20. Janowczyk, A., Doyle, S., Gilmore, H., et al: ‘A resolution adaptive deep hierarchical (RADHical) learning scheme applied to nuclear segmentation of digital pathology images’, Comput. Methods Biomech. Biomed. Eng. Imaging Vis., 2018, 6, (3), pp. 270–276.
    63. 63)
      • 24. Li, W., Manivannan, S., Akbar, S., et al: ‘Gland segmentation in colon histology images using hand-crafted features and convolutional neural networks’. IEEE 13th Int. Symp. on Biomedical Imaging (ISBI), Prague, Czech Republic, 2016, pp. 1405–1408.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.6513
Loading

Related content

content/journals/10.1049/iet-ipr.2018.6513
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading