Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free DeepInterpolation: fusion of multiple interpolations and CNN to obtain super-resolution

© The Institution of Engineering and Technology
Received 20/09/2019, Accepted 30/11/2020, Revised 27/11/2020, Published 14/12/2020

The authors propose architectures that learn end-to-end mapping functions to improve the spatial resolution of the input natural images. The models are unique in forming non-linear combinations of three image interpolation techniques using the convolutional neural network. Another proposed architecture uses a skip connection with nearest-neighbour interpolation, achieving almost similar results. The architectures have been carefully designed to ensure that the reconstructed images lie precisely in the manifold of high-resolution images, thereby preserving the high-frequency components with fine details. They have compared with the state-of-the-art and recent deep learning-based natural image super-resolution techniques and found that their methods can preserve the sharp details in the image, while also obtaining comparable or better peak-signal-to-noise ratio values than them. Since their methods use image interpolations and a shallow convolutional neural network (CNN) with a fewer number of smaller filters, the computational cost is kept low. They have reported the results of the best two proposed architectures on five standard data sets for an upscale factor of 2. Their methods generalise well in most cases, which is evident from the better results obtained with increasingly complex data sets. For four times upscaling, they have designed similar architectures for comparing with other methods.

Inspec keywords: image reconstruction; learning (artificial intelligence); image resolution; interpolation; image fusion; image filtering; convolutional neural nets

Other keywords: nearest-neighbour interpolation; image interpolations; spatial resolution; input natural images; recent deep learning-based natural image super-resolution techniques; image interpolation techniques; end-to-end mapping functions; high-frequency components; shallow convolutional neural network; similar architectures; reconstructed images; high-resolution images; skip connection; multiple interpolations; peak-signal-to-noise ratio values; nonlinear combinations; CNN

Subjects: Filtering methods in signal processing; Neural nets; Optical, image and video signal processing; Computer vision and image processing techniques; Interpolation and function approximation (numerical analysis); Machine learning (artificial intelligence); Interpolation and function approximation (numerical analysis)

References

    1. 1)
      • 36. Huang, J.-B., Singh, A., Ahuja, N.: ‘Single image super-resolution from transformed self-exemplars’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2015, pp. 5197–5206.
    2. 2)
      • 28. Pandey, R.K., Karmakar, S., Ramakrishnan, A.G., et al: ‘Improving facial emotion recognition systems with crucial feature extractors’. Int. Conf. on Image Analysis and Processing, Trento, Italy, 2019, pp. 268–279.
    3. 3)
      • 31. Pandey, R.K., Vignesh, K., Ramakrishnan, A.G., et al: ‘Binary document image super resolution for improved readability and OCR performance’, arXiv preprint arXiv:1812.02475, 2018.
    4. 4)
      • 4. Parker, J., Anthony, R.V.K., Troxel, D.E.: ‘Comparison of interpolating methods for image resampling’, IEEE Trans. Med. Imaging, 1983, 2, (1), pp. 31–39.
    5. 5)
      • 12. Timofte, R., De Smet, V., Gool, L.V.: ‘Anchored neighborhood regression for fast example-based super-resolution’. Proc. IEEE Int. Conf. on Computer Vision, Sydney, Australia, 2013.
    6. 6)
      • 13. Timofte, R., De Smet, V., Gool, L.V.: ‘A+: adjusted anchored neighborhood regression for fast super-resolution’. Asian Conf. on Computer Vision, Singapore, 2014.
    7. 7)
      • 32. Lai, W.S., Huang, J.B., Ahuja, N., et al: ‘Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks’, arXiv preprint arXiv:1710.01992, 4 October 2017.
    8. 8)
      • 26. Dai, S., Han, M., Xu, W., et al: ‘Softcuts: a soft edge smoothness prior for color image superresolution’, IEEE Trans. Image Process., 2009, 18, (5), pp. 969–981.
    9. 9)
      • 40. Glasner, D., Bagon, S., Irani, M.: ‘Super-resolution from a single image’. IEEE 12th Int. Conf. on Computer Vision, Kyoto, Japan, 2009.
    10. 10)
      • 18. Lai, W.-S., Huang, J.-B., Ahuja, N., et al: ‘Deep Laplacian pyramid networks for fast and accurate super-resolution’. Proc. IEEE Conf. on Computer Vision Pattern Recognition, Honolulu, HI, USA, 2017.
    11. 11)
      • 21. Ledig, C., Theis, L., Huszár, F., et al: ‘Photo-realistic single image super-resolution using a generative adversarial network’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017.
    12. 12)
      • 10. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution via sparse representation’, IEEE Trans. Image Process., 2010, 19, (11), pp. 2861–2873.
    13. 13)
      • 22. Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al: ‘Generative adversarial nets’. Advances in Neural Information Processing Systems (NIPS), Montreal, Canada, 2014, pp. 2672–2680.
    14. 14)
      • 5. Giachetti, A., Asuni, N.: ‘Real-time artifact-free image upscaling’, IEEE Trans. Image Process., 2011, 20, (10), pp. 2760–2768.
    15. 15)
      • 20. Johnson, J., Alahi, A., Li, F.: ‘Perceptual losses for real-time style transfer and super-resolution’, European Conf. on Computer Vision (ECCV), Amsterdam, Netherlands, 2016, pp. 694–711.
    16. 16)
      • 1. Dong, C., Loy, C.C., He, K., et al: ‘Learning a deep convolutional network for image super-resolution’. European Conf. on Computer Vision, Zurich, Switzerland, 2014.
    17. 17)
      • 14. Dong, C., Loy, C.C., He, K., et al: ‘Image super-resolution using deep convolutional networks’, IEEE Trans. Pattern Anal. Mach. Intell., 2016, 38, (2), pp. 295–307.
    18. 18)
      • 29. Pandey, R.K., Ramakrishnan, A.G., Karmakar, S.: ‘Effects of modifying the input features and the loss function on improving emotion classification’. IEEE Region 10 Conference (TENCON), Kochi, India, 2019, pp. 1159–1162.
    19. 19)
      • 34. Zeyde, R., Elad, M., Protter, M.: ‘On single image scale-up using sparse-representations’. Proc. Int. Conf. on Curves and Surfaces, Avignon-France, 24–30 June 2010.
    20. 20)
      • 15. Dong, C., Loy, C.C., Tang, X: ‘Accelerating the super-resolution convolutional neural network’. European Conf. on Computer Vision, Amsterdam, Netherlands, 2016.
    21. 21)
      • 6. Giachetti, A., Asuni, N.: ‘Fast artifacts-free image interpolation’. British Machine Vision Conf. (BMVC), Leeds, UK, 2008.
    22. 22)
      • 23. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’. Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, 2015.
    23. 23)
      • 25. Shi, W., Caballero, J., Huszár, F., et al: ‘Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016.
    24. 24)
      • 37. Agustsson, E., Timofte, R.: ‘NTIRE 2017 challenge on single image super-resolution: dataset and study’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops, Honolulu, HI, USA, July 2017.
    25. 25)
      • 2. Keys, R.: ‘Cubic convolution interpolation for digital image processing’, IEEE Trans. Acoust. Speech Signal Process., 1981, 29, (6), pp. 1153–1160.
    26. 26)
      • 11. Yang, J., Wang, Z., Lin, Z., et al: ‘Coupled dictionary training for image super-resolution’, IEEE Trans. Image Process., 2012, 21, (8), pp. 3467–3478.
    27. 27)
      • 35. Martin, D., Fowlkes, C., Tal, D., et al: ‘A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics’. Proc. 8th Int. Conf. on Computer Vision,vol. 2, Vancouver, Canada, July 2001, pp. 416–423.
    28. 28)
      • 19. Xu, L., Ren, J.S.J., Liu, C., et al: ‘Deep convolutional neural network for image deconvolution’. Advances Neural Information Processing Systems, Montreal, Canada, 2014.
    29. 29)
      • 17. Kim, J., Lee, J.K., Lee, K.M.: ‘Deeply-recursive convolutional network for image super-resolution’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016.
    30. 30)
      • 27. Kim, K.I., Kwon, Y.: ‘Single-image super-resolution using sparse regression and natural image prior’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (6), pp. 1127–1133.
    31. 31)
      • 38. Kinga, D., Ba, J.: ‘Adam: a method for stochastic optimization’. Proc. Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, 2015.
    32. 32)
      • 16. Kim, J., Lee, J.K., Lee, K.M.: ‘Accurate image super-resolution using very deep convolutional networks’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, , 2016.
    33. 33)
      • 9. Freeman, W.T., Jones, T.R., Pasztor, E.C.: ‘Example-based super-resolution’, IEEE Comput. Graphics Appl., 2002, 22, (2), pp. 56–65.
    34. 34)
      • 30. Pandey, R.K., Maiya, S.R., Ramakrishnan, A.G.: ‘A new approach for upscaling document images for improving their quality’. 14th IEEE India Council Int. Conf. (INDICON), Roorkee, India, 2017.
    35. 35)
      • 41. He, K., Sun, J.: ‘Convolutional neural networks at constrained time cost’. Proc. IEEE Conf. on Computer Vision Pattern Recognition, Boston, MA, USA, 2015.
    36. 36)
      • 33. Bevilacqua, M., Roumy, A., Guillemot, C., et al: ‘Low-complexity single-image super-resolution based on nonnegative neighbor embedding’'. Proc. British Machine Vision Conf. (BMVC), Surrey, Canada, 2012.
    37. 37)
      • 7. Yang, C.-Y., Ma, C., Yang, M.-H.: ‘Single-image super-resolution: a benchmark’. European Conf. on Computer Vision, Zurich, Switzerland, 2014.
    38. 38)
      • 8. Chang, H., Yeung, D.-Y., Xiong, Y.: ‘Super-resolution through neighbor embedding’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Washington, DC, USA, vol. 1, 2004.
    39. 39)
      • 39. Reddi, S.J., Kale, S., Kumar, S.: ‘On the convergence of Adam and beyond’. Proc. Int. Conf. on Learning Representations (ICLR), Vancouver, Canada, 2018.
    40. 40)
      • 24. Akilan, T., Jonathan Wu, Q.M., Zhang, W.: ‘Video foreground extraction using multi-view receptive field and encoder–decoder DCNN for traffic and surveillance applications’, IEEE Trans. Veh. Technol., 2019, 68, (10), pp. 9478–9493.
    41. 41)
      • 3. Meijering, E.: ‘A chronology of interpolation: from ancient astronomy to modern signal and image process’, Proc. IEEE, 2002, 90, (3), pp. 319–342.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2019.1244
Loading

Related content

content/journals/10.1049/iet-ipr.2019.1244
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address