access icon free Spatialising uncertainty in image segmentation using weakly supervised convolutional neural networks: a case study from historical map processing

Convolutional neural networks (CNNs) such as encoder–decoder CNNs have increasingly been employed for semantic image segmentation at the pixel-level requiring pixel-level training labels, which are rarely available in real-world scenarios. In practice, weakly annotated training data at the image patch level are often used for pixel-level segmentation tasks, requiring further processing to obtain accurate results, mainly because the translation invariance of the CNN-based inference can turn into an impeding property leading to segmentation results of coarser spatial granularity compared with the original image. However, the inherent uncertainty in the segmented image and its relationships to translation invariance, CNN architecture, and classification scheme has never been analysed from an explicitly spatial perspective. Therefore, the authors propose measures to spatially visualise and assess class decision confidence based on spatially dense CNN predictions, resulting in continuous decision confidence surfaces. They find that such a visual-analytical method contributes to a better understanding of the spatial variability of class score confidence derived from weakly supervised CNN-based classifiers. They exemplify this approach by incorporating decision confidence surfaces into a processing chain for the extraction of human settlement features from historical map documents based on weakly annotated training data using different CNN architectures and classification schemes.

Inspec keywords: image resolution; document image processing; image classification; feedforward neural nets; image segmentation; history; feature extraction; inference mechanisms; cartography; data visualisation

Other keywords: weakly supervised CNN-based classifiers; image patch level; weakly annotated training data; weakly supervised convolutional neural networks; visual-analytical method; class decision confidence; human settlement feature extraction; semantic image segmentation; spatialising uncertainty; pixel-level training labels; CNN-based inference; spatial granularity; continuous decision confidence surfaces; pixel-level segmentation tasks; classification scheme; class score confidence spatial variability; translation invariance; historical map documents; encoder-decoder CNN; historical map processing

Subjects: Document processing and analysis techniques; Knowledge engineering techniques; Image recognition; Geography and cartography computing; Computer vision and image processing techniques; Graphics techniques; Neural computing techniques

References

    1. 1)
      • 36. Rosenfield, G., Melley, M.: ‘Applications of statistics to thematic mapping’, Photogramm. Eng. Remote Sens., 1980, 46, pp. 12871294.
    2. 2)
      • 14. Maire, F., Mejias, L., Hodgson, A.: ‘A convolutional neural network for automatic analysis of aerial imagery’. IEEE Int. Conf. on Digital Image Computing: Techniques and Applications (DICTA), Wollongong, Australia, 2014, pp. 18.
    3. 3)
      • 4. Cordts, M., Omran, M., Ramos, S., et al: ‘The cityscapes dataset for semantic urban scene understanding’. Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016.
    4. 4)
      • 2. Ball, J. E., Anderson, D. T., Chan, C. S.: ‘Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community’, J. Appl. Remote Sens., 2017, 11, (4), p. 042609.
    5. 5)
      • 9. Durand, T., Mordan, T., Thome, N., et al: ‘Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation’. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, Hawaii, USA, 2017.
    6. 6)
      • 35. Forbes, A.D.: ‘Classification algorithm evaluation: five performance measures based on confusion matrices’, J. Clin. Monit. Comput., 1995, 11, pp. 189206.
    7. 7)
      • 19. Chiang, Y.-Y., Leyk, S., Knoblock, C.A.: ‘A survey of digital map processing techniques’, ACM Comput. Surv. (CSUR), 2014, 47, (1), p. 1.
    8. 8)
      • 8. Papandreou, G., Chen, L. C., Murphy, K. P., et al: ‘Weakly- and semi-supervised learning of a deep convolutional network for semantic image segmentation’. Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 17421750.
    9. 9)
      • 11. Zhao, W., Jiao, L., Ma, W., et al: ‘Superpixel-based multiple local CNN for panchromatic and multispectral image classification’, IEEE Trans. Geosci. Remote Sens., 2017, 55, (7), pp. 41414156.
    10. 10)
      • 25. Yu, R., Luo, Z., Chiang, Y.-Y.: ‘Recognizing text in historical maps using maps from multiple time periods’. Proc. IEEE 23rd Int. Conf. on Pattern Recognition (ICPR), Cancun, Mexico, 2016, pp. 39933998.
    11. 11)
      • 12. Kauderer-Abrams, E.: ‘Quantifying translation-invariance in convolutional neural networks’, 2016. Available at http://cs231n.stanford.edu/reports/2016/pdfs/107_Report.pdf, accessed 8 January 2018.
    12. 12)
      • 26. Fishburn, K.A., Davis, L.R., Allord, G.J.: ‘Scanning and georeferencing historical USGS quadrangles’, U.S. Geol. Surv. fact sheet, 2017, 3048, pp. 12, https://doi.org/10.3133/fs20173048.
    13. 13)
      • 5. Zhou, Z. H.: ‘A brief introduction to weakly supervised learning’, Nat. Sci. Rev., 2018, 5, (1), pp. 4453.
    14. 14)
      • 37. Fawcett, T.: ‘An introduction to ROC analysis’, Pattern Recognit. Lett., 2005, 27, (8), pp. 861874.
    15. 15)
      • 24. Chiang, Y.-Y., Leyk, S., Nazari, N.H., et al: ‘Assessing impact of graphical quality on automatic text recognition in digital maps’, Comput. Geosci., 2016, 93, pp. 2135.
    16. 16)
      • 13. Audebert, N., Le Saux, B., Lefèvre, S.: ‘Semantic segmentation of earth observation data using multimodal and multi-scale deep networks’. Proc. Asian Conf. on Computer Vision, Taipei, Taiwan, 2016, pp. 180196.
    17. 17)
      • 20. Uhl, J. H., Leyk, S., Chiang, Y.-Y., et al: ‘Map archive mining: visual-analytical approaches to explore large historical map collections’, ISPRS Int. J. Geo-Inf., 2018, 7, p. 148.
    18. 18)
      • 23. Chiang, Y.-Y., Leyk, S.: ‘Exploiting online gazetteer for fully automatic extraction of cartographic symbols’. Proc. 27th Int. Cartographic Conf. (ICC), Rio de Janeiro, Brazil, 2015.
    19. 19)
      • 21. Duan, W., Chiang, Y. Y., Knoblock, C. A., et al: ‘Automatic alignment of geographic features in contemporary vector data and historical maps’. Proc. First Workshop on Artificial Intelligence and Deep Learning for Geographic Knowledge Discovery, Redondo Beach, California, USA, 2017.
    20. 20)
      • 33. Chatfield, K., Simonyan, K., Vedaldi, A., et al: ‘Return of the devil in the details: delving deep into convolutional nets’. British Machine Vision Conf., Nottingham, UK, 2014, arXiv ref. cs1405.3531.
    21. 21)
      • 27. Library of Congress: ‘Geography and map division’. 2018. Available at https://www.loc.gov/collections/sanborn-maps/, accessed 8 January 2018.
    22. 22)
      • 22. Leyk, S., Chiang, Y.-Y.: ‘Information extraction based on the concept of geographic context’. Proc. AutoCarto, 2016, Albuquerque, NM, USA, 14–16 September 2016, 2016, pp. 100110.
    23. 23)
      • 6. Badrinarayanan, V., Kendall, A., Cipolla, R.: ‘Segnet: a deep convolutional encoder-decoder architecture for image segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39–12, pp. 24812495.
    24. 24)
      • 16. Marmanis, D., Datcu, M., Esch, T., et al: ‘Deep learning earth observation classification using ImageNet pretrained networks’, IEEE Geosci. Remote Sens. Lett., 2016, 13, (1), pp. 105109.
    25. 25)
      • 7. Long, J., Shelhamer, E., Darrell, T.: ‘Fully convolutional networks for semantic segmentation’. Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, Massachusetts, USA, 2015, pp. 34313440.
    26. 26)
      • 30. Lowe, D.G.: ‘Object recognition from local scale-invariant features’. Proc. Seventh IEEE Int. Conf. on Computer Vision, Kerkyra, Greece, 1999, pp. 11501157.
    27. 27)
      • 31. Maaten, L. V. D., Hinton, G.: ‘Visualizing data using t-SNE’, J. Mach. Learn. Res., 2008, 9, pp. 25792605.
    28. 28)
      • 10. Chen, L.C., Papandreou, G., Kokkinos, I., et al: ‘Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs’. 2016, arXiv preprint arXiv:1606.00915. DOI: 10.1109/TPAMI.2017.2699184.
    29. 29)
      • 3. Rottensteiner, F., Sohn, G., Jung, J., et al: ‘The ISPRS benchmark on urban object classification and 3D building reconstruction’, ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci, 2012, 1, (3), pp. 293298.
    30. 30)
      • 28. Uhl, J. H., Leyk, S., Chiang, Y.-Y., et al: ‘Extracting human settlement footprint from historical topographic map series using context-based machine learning’. Proc. Eighth Int. Conf. on Pattern Recognition Systems (ICPRS 2017), Madrid, Spain, 2017, pp. 1521.
    31. 31)
      • 38. Leyk, S., Zimmermann, N.E.: ‘A predictive uncertainty model for field-based survey maps using generalized linear models’, in Egenhofer M., Freksa C., Miller H. (Eds.): Proc. Third Int. Conf. on Geographic Information Science (GIScience, 2004), Adelphi, MD, USA, 20–23 October 2004, (LNCS, 3234), pp. 191205.
    32. 32)
      • 29. Zillow Transaction and Assessment Dataset (ZTRAX): ‘Available through a data use agreement between the university of Colorado Boulder and Zillow Inc.’, 2016.
    33. 33)
      • 1. Zhang, L., Zhang, L., Du, B.: ‘Deep learning for remote sensing data: a technical tutorial on the state of the art’, IEEE Geosci. Remote Sens. Mag., 2016, 4, (2), pp. 2240.
    34. 34)
      • 18. Scott, G. J., England, M.R., Starms, W.A., et al: ‘Training deep convolutional neural networks for land-cover classification of high-resolution imagery’, IEEE Geosci. Remote Sens. Lett., 2017, 14, (4), pp. 549553.
    35. 35)
      • 32. LeCun, Y., Boser, B., Denker, J.S., et al: ‘Backpropagation applied to handwritten zip code recognition’, Neural Comput., 1989, 1, (4), pp. 541551.
    36. 36)
      • 17. Romero, A., Gatta, C., Camps-Valls, G.: ‘Unsupervised deep feature extraction for remote sensing image classification’, IEEE Trans. Geosci. Remote Sens., 2016, 54, (3), pp. 13491362.
    37. 37)
      • 34. Cohen, J.: ‘A coefficient of agreement for nominal scales’, Educ. Psychol. Meas., 1960, 20, pp. 3746.
    38. 38)
      • 15. Castelluccio, M., Poggi, G., Sansone, C., et al: ‘Land use classification in remote sensing images by convolutional neural networks’. 2015, arXiv preprint arXiv:1508.00092.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.5484
Loading

Related content

content/journals/10.1049/iet-ipr.2018.5484
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading