Spatialising uncertainty in image segmentation using weakly supervised convolutional neural networks: a case study from historical map processing

Spatialising uncertainty in image segmentation using weakly supervised convolutional neural networks: a case study from historical map processing

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for $120.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Image Processing — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Convolutional neural networks (CNNs) such as encoder–decoder CNNs have increasingly been employed for semantic image segmentation at the pixel-level requiring pixel-level training labels, which are rarely available in real-world scenarios. In practice, weakly annotated training data at the image patch level are often used for pixel-level segmentation tasks, requiring further processing to obtain accurate results, mainly because the translation invariance of the CNN-based inference can turn into an impeding property leading to segmentation results of coarser spatial granularity compared with the original image. However, the inherent uncertainty in the segmented image and its relationships to translation invariance, CNN architecture, and classification scheme has never been analysed from an explicitly spatial perspective. Therefore, the authors propose measures to spatially visualise and assess class decision confidence based on spatially dense CNN predictions, resulting in continuous decision confidence surfaces. They find that such a visual-analytical method contributes to a better understanding of the spatial variability of class score confidence derived from weakly supervised CNN-based classifiers. They exemplify this approach by incorporating decision confidence surfaces into a processing chain for the extraction of human settlement features from historical map documents based on weakly annotated training data using different CNN architectures and classification schemes.


    1. 1)
      • 1. Zhang, L., Zhang, L., Du, B.: ‘Deep learning for remote sensing data: a technical tutorial on the state of the art’, IEEE Geosci. Remote Sens. Mag., 2016, 4, (2), pp. 2240.
    2. 2)
      • 2. Ball, J. E., Anderson, D. T., Chan, C. S.: ‘Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community’, J. Appl. Remote Sens., 2017, 11, (4), p. 042609.
    3. 3)
      • 3. Rottensteiner, F., Sohn, G., Jung, J., et al: ‘The ISPRS benchmark on urban object classification and 3D building reconstruction’, ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci, 2012, 1, (3), pp. 293298.
    4. 4)
      • 4. Cordts, M., Omran, M., Ramos, S., et al: ‘The cityscapes dataset for semantic urban scene understanding’. Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016.
    5. 5)
      • 5. Zhou, Z. H.: ‘A brief introduction to weakly supervised learning’, Nat. Sci. Rev., 2018, 5, (1), pp. 4453.
    6. 6)
      • 6. Badrinarayanan, V., Kendall, A., Cipolla, R.: ‘Segnet: a deep convolutional encoder-decoder architecture for image segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39–12, pp. 24812495.
    7. 7)
      • 7. Long, J., Shelhamer, E., Darrell, T.: ‘Fully convolutional networks for semantic segmentation’. Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, Massachusetts, USA, 2015, pp. 34313440.
    8. 8)
      • 8. Papandreou, G., Chen, L. C., Murphy, K. P., et al: ‘Weakly- and semi-supervised learning of a deep convolutional network for semantic image segmentation’. Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 17421750.
    9. 9)
      • 9. Durand, T., Mordan, T., Thome, N., et al: ‘Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation’. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, Hawaii, USA, 2017.
    10. 10)
      • 10. Chen, L.C., Papandreou, G., Kokkinos, I., et al: ‘Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs’. 2016, arXiv preprint arXiv:1606.00915. DOI: 10.1109/TPAMI.2017.2699184.
    11. 11)
      • 11. Zhao, W., Jiao, L., Ma, W., et al: ‘Superpixel-based multiple local CNN for panchromatic and multispectral image classification’, IEEE Trans. Geosci. Remote Sens., 2017, 55, (7), pp. 41414156.
    12. 12)
      • 12. Kauderer-Abrams, E.: ‘Quantifying translation-invariance in convolutional neural networks’, 2016. Available at, accessed 8 January 2018.
    13. 13)
      • 13. Audebert, N., Le Saux, B., Lefèvre, S.: ‘Semantic segmentation of earth observation data using multimodal and multi-scale deep networks’. Proc. Asian Conf. on Computer Vision, Taipei, Taiwan, 2016, pp. 180196.
    14. 14)
      • 14. Maire, F., Mejias, L., Hodgson, A.: ‘A convolutional neural network for automatic analysis of aerial imagery’. IEEE Int. Conf. on Digital Image Computing: Techniques and Applications (DICTA), Wollongong, Australia, 2014, pp. 18.
    15. 15)
      • 15. Castelluccio, M., Poggi, G., Sansone, C., et al: ‘Land use classification in remote sensing images by convolutional neural networks’. 2015, arXiv preprint arXiv:1508.00092.
    16. 16)
      • 16. Marmanis, D., Datcu, M., Esch, T., et al: ‘Deep learning earth observation classification using ImageNet pretrained networks’, IEEE Geosci. Remote Sens. Lett., 2016, 13, (1), pp. 105109.
    17. 17)
      • 17. Romero, A., Gatta, C., Camps-Valls, G.: ‘Unsupervised deep feature extraction for remote sensing image classification’, IEEE Trans. Geosci. Remote Sens., 2016, 54, (3), pp. 13491362.
    18. 18)
      • 18. Scott, G. J., England, M.R., Starms, W.A., et al: ‘Training deep convolutional neural networks for land-cover classification of high-resolution imagery’, IEEE Geosci. Remote Sens. Lett., 2017, 14, (4), pp. 549553.
    19. 19)
      • 19. Chiang, Y.-Y., Leyk, S., Knoblock, C.A.: ‘A survey of digital map processing techniques’, ACM Comput. Surv. (CSUR), 2014, 47, (1), p. 1.
    20. 20)
      • 20. Uhl, J. H., Leyk, S., Chiang, Y.-Y., et al: ‘Map archive mining: visual-analytical approaches to explore large historical map collections’, ISPRS Int. J. Geo-Inf., 2018, 7, p. 148.
    21. 21)
      • 21. Duan, W., Chiang, Y. Y., Knoblock, C. A., et al: ‘Automatic alignment of geographic features in contemporary vector data and historical maps’. Proc. First Workshop on Artificial Intelligence and Deep Learning for Geographic Knowledge Discovery, Redondo Beach, California, USA, 2017.
    22. 22)
      • 22. Leyk, S., Chiang, Y.-Y.: ‘Information extraction based on the concept of geographic context’. Proc. AutoCarto, 2016, Albuquerque, NM, USA, 14–16 September 2016, 2016, pp. 100110.
    23. 23)
      • 23. Chiang, Y.-Y., Leyk, S.: ‘Exploiting online gazetteer for fully automatic extraction of cartographic symbols’. Proc. 27th Int. Cartographic Conf. (ICC), Rio de Janeiro, Brazil, 2015.
    24. 24)
      • 24. Chiang, Y.-Y., Leyk, S., Nazari, N.H., et al: ‘Assessing impact of graphical quality on automatic text recognition in digital maps’, Comput. Geosci., 2016, 93, pp. 2135.
    25. 25)
      • 25. Yu, R., Luo, Z., Chiang, Y.-Y.: ‘Recognizing text in historical maps using maps from multiple time periods’. Proc. IEEE 23rd Int. Conf. on Pattern Recognition (ICPR), Cancun, Mexico, 2016, pp. 39933998.
    26. 26)
      • 26. Fishburn, K.A., Davis, L.R., Allord, G.J.: ‘Scanning and georeferencing historical USGS quadrangles’, U.S. Geol. Surv. fact sheet, 2017, 3048, pp. 12,
    27. 27)
      • 27. Library of Congress: ‘Geography and map division’. 2018. Available at, accessed 8 January 2018.
    28. 28)
      • 28. Uhl, J. H., Leyk, S., Chiang, Y.-Y., et al: ‘Extracting human settlement footprint from historical topographic map series using context-based machine learning’. Proc. Eighth Int. Conf. on Pattern Recognition Systems (ICPRS 2017), Madrid, Spain, 2017, pp. 1521.
    29. 29)
      • 29. Zillow Transaction and Assessment Dataset (ZTRAX): ‘Available through a data use agreement between the university of Colorado Boulder and Zillow Inc.’, 2016.
    30. 30)
      • 30. Lowe, D.G.: ‘Object recognition from local scale-invariant features’. Proc. Seventh IEEE Int. Conf. on Computer Vision, Kerkyra, Greece, 1999, pp. 11501157.
    31. 31)
      • 31. Maaten, L. V. D., Hinton, G.: ‘Visualizing data using t-SNE’, J. Mach. Learn. Res., 2008, 9, pp. 25792605.
    32. 32)
      • 32. LeCun, Y., Boser, B., Denker, J.S., et al: ‘Backpropagation applied to handwritten zip code recognition’, Neural Comput., 1989, 1, (4), pp. 541551.
    33. 33)
      • 33. Chatfield, K., Simonyan, K., Vedaldi, A., et al: ‘Return of the devil in the details: delving deep into convolutional nets’. British Machine Vision Conf., Nottingham, UK, 2014, arXiv ref. cs1405.3531.
    34. 34)
      • 34. Cohen, J.: ‘A coefficient of agreement for nominal scales’, Educ. Psychol. Meas., 1960, 20, pp. 3746.
    35. 35)
      • 35. Forbes, A.D.: ‘Classification algorithm evaluation: five performance measures based on confusion matrices’, J. Clin. Monit. Comput., 1995, 11, pp. 189206.
    36. 36)
      • 36. Rosenfield, G., Melley, M.: ‘Applications of statistics to thematic mapping’, Photogramm. Eng. Remote Sens., 1980, 46, pp. 12871294.
    37. 37)
      • 37. Fawcett, T.: ‘An introduction to ROC analysis’, Pattern Recognit. Lett., 2005, 27, (8), pp. 861874.
    38. 38)
      • 38. Leyk, S., Zimmermann, N.E.: ‘A predictive uncertainty model for field-based survey maps using generalized linear models’, in Egenhofer M., Freksa C., Miller H. (Eds.): Proc. Third Int. Conf. on Geographic Information Science (GIScience, 2004), Adelphi, MD, USA, 20–23 October 2004, (LNCS, 3234), pp. 191205.

Related content

This is a required field
Please enter a valid email address