access icon openaccess Image super-resolution based on conditional generative adversarial network

This is an open access article published by the IET under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/)
Received 28/06/2018, Accepted 24/03/2020, Revised 26/02/2020, Published 26/05/2020

Generative adversarial network (GAN) is one of the most prevalent generative models that can synthesise realistic high-frequency details. However, a mismatch between the input and the output may arise when GAN is directly applied to image super-resolution. To alleviate this issue, the authors adopted a conditional GAN (cGAN) in this study. The cGAN discriminator attempted to guess whether the unknown high-resolution (HR) image was produced by the generator with the aid of the original low-resolution (LR) image. They propose a novel discriminator that only penalises at the scale of the patch and, thus, has relatively few parameters to train. The generator of cGAN is an encoder–decoder with skip connections to shuttle the shared low-level information directly across the network. To better maintain the low-frequency information and recover the high-frequency information, they designed a generator loss function combining adversarial loss term and L1 loss term. The former term is beneficial to the synthesis of fine-grained textures, while the latter is responsible for learning the overall structure of the LR input. The experiments revealed that the proposed method could generate HR images with richer details and less over-smoothness.

Inspec keywords: neural nets; image resolution; unsupervised learning

Other keywords: L1 loss term; conditional generative adversarial network; image super-resolution; adversarial loss term; low-resolution image; HR images; high-resolution image; generator loss function; conditional GAN; high-frequency information; cGAN discriminator; low-frequency information

Subjects: Computer vision and image processing techniques; Neural computing techniques; Optical, image and video signal processing

References

    1. 1)
      • 1. Nasrollahi, K., Moeslund, B.: ‘Super-resolution: a comprehensive survey’, Mach. Vis. Appl., 2014, 25, (6), pp. 1423–1468.
    2. 2)
      • 17. Shi, W., Caballero, J., Huszár, F., et al: ‘Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 1874–1883.
    3. 3)
      • 5. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution via sparse representation’, IEEE Trans. Image Process., 2010, 19, (11), pp. 2861–2873.
    4. 4)
      • 9. Yang, C.Y., Yang, M.H.: ‘Fast direct super-resolution by simple functions’. Proc. of the IEEE Int. Conf. on Computer Vision (ICCV), Sydney, Australia, 2013, pp. 561–568.
    5. 5)
      • 10. Zhang, Z., Li, F., Zhao, M., et al: ‘Robust neighborhood preserving projection by nuclear/L2, 1-norm regularization for image feature extraction’, IEEE Trans. Image Process., 2017, 26, (4), pp. 1607–1622.
    6. 6)
      • 12. Dong, C., Chen, C.L., He, K., et al: ‘Image super-resolution using deep convolutional networks’, IEEE Trans. Pattern Anal. Mach. Intell., 2016, 38, (2), p. 295.
    7. 7)
      • 6. Li, J., Yuan, Q., Shen, H., et al: ‘Hyperspectral image super-resolution by spectral mixture analysis and spatial–spectral group sparsity’, IEEE Geosci. Remote Sens., 2016, 13, (9), pp. 1250–1254.
    8. 8)
      • 15. Wei, Y., Yuan, Q., Shen, H., et al: ‘Boosting the accuracy of multispectral image pan-sharpening by learning a deep residual network’, IEEE Geosci. Remote Sens., 2017, 14, (10), pp. 1795–1799.
    9. 9)
      • 19. Metz, L., Poole, B., Pfau, D., et al: ‘Unrolled generative adversarial networks’. arXiv:1611.02163, 2016.
    10. 10)
      • 7. Lorenzi, L., Melgani, F., Mercier, G.: ‘Missing-area reconstruction in multispectral images under a compressive sensing perspective’, IEEE Trans. Geosci. Remote Sens., 2013, 51, (7), pp. 3998–4008.
    11. 11)
      • 23. Ioffe, S., Szegedy, C.: ‘Batch normalization: accelerating deep network training by reducing internal covariate shift’. Int. Conf. on Machine Learning (ICML), Lille, France, 2015, pp. 448–456.
    12. 12)
      • 16. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 770–778.
    13. 13)
      • 11. Zhang, Z., Ren, J., Li, S., et al: ‘Robust subspace discovery by block-diagonal adaptive locality-constrained representation’. arXiv: 1908.01266, 2019.
    14. 14)
      • 27. Kingma, D.P., Ba, J.: ‘Adam: a method for stochastic optimization’. arXiv:1412.6980, 2014.
    15. 15)
      • 25. Denton, E., Chintala, S., Szlam, A., et al: ‘Deep generative image models using a Laplacian pyramid of adversarial networks’. Advances in Neural Information Processing Systems (NIPS), Montreal, Quebec, Canada, 2015, pp. 1486–1494.
    16. 16)
      • 2. Li, X., Shen, H., Zhang, L., et al: ‘Recovering quantitative remote sensing products contaminated by thick clouds and shadows using multitemporal dictionary learning’, IEEE Trans. Geosci. Remote Sens., 2014, 52, (11), pp. 7086–7098.
    17. 17)
      • 20. Pathak, D., Krahenbuhl, P., Donahue, J., et al: ‘Context encoders: feature learning by inpainting’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 2536–2544.
    18. 18)
      • 29. Haris, M., Shakhnarovich, G., Ukita, N.: ‘Deep back-projection networks for super-resolution’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018, pp. 1664–1673.
    19. 19)
      • 28. Zhang, K., Zuo, W., Zhang, L.: ‘Learning a single convolutional super-resolution network for multiple degradations’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018, pp. 3262–3271.
    20. 20)
      • 22. Johnson, J., Alahi, A., Li, F.F.: ‘Perceptual losses for real-time style transfer and super-resolution’. Euro. Conf. on Computer Vision (ECCV), Amsterdam, The Netherlands, 2016, pp. 694–711.
    21. 21)
      • 8. Timofte, R., De, S.V., Van, G.L.: ‘A + : adjusted anchored neighbourhood regression for fast super-resolution’. Asian Conf. on Computer Vision (ACCV), Singapore, 2014, pp. 111–126.
    22. 22)
      • 14. Zhang, Q., Yuan, Q., Zeng, C., et al: ‘Missing data reconstruction in remote sensing image with a unified spatial–temporal–spectral deep convolutional neural network’. IEEE Trans. Geosci. Remote Sens., 2016, 56, (8), pp. 4274–4288.
    23. 23)
      • 3. Li, X., Shen, H., Zhang, L., et al: ‘Sparse-based reconstruction of missing information in remote sensing images from spectral/temporal complementary information’, ISPRS J. Photogramm. Remote Sens., 2015, 106, pp. 1–15.
    24. 24)
      • 24. Goodfellow, I.J., Pouget, A.J., Mirza, M., et al: ‘Generative adversarial nets’. Advances in Neural Information Processing Systems (NIPS), Montreal, Quebec, Canada, 2014, pp. 2672–2680.
    25. 25)
      • 26. Lin, T.Y., Maire, M., Belongie, S., et al: ‘Microsoft COCO: common objects in context’. Euro. Conf. on Computer Vision (ECCV), Zurich, Switzerland, 2014, pp. 740–755.
    26. 26)
      • 30. Wang, Z., Bovik, A.C., Simoncelli, E.P.: ‘Structural approaches to image quality assessment’, in ‘Handbook of image and video processing’ (Academic Press, Cambridge, MA, USA, 2005), p. 18.
    27. 27)
      • 18. Ledig, C., Theis, L., Huszar, F., et al: ‘Photo-realistic single image super-resolution using a generative adversarial network’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Puerto Rico, 2017, pp. 105–114.
    28. 28)
      • 21. Wang, X., Gupta, A.: ‘Generative image modelling using style and structure adversarial networks’. Euro. Conf. on Computer Vision (ECCV), Amsterdam, The Netherlands, 2016, pp. 318–335.
    29. 29)
      • 4. Yang, J., Wright, J., Huang, T., et al: ‘Image super-resolution as sparse representation of raw image patches’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2008, pp. 1–8.
    30. 30)
      • 13. Kim, J., Lee, J.K., Lee, K.M.: ‘Accurate image super-resolution using very deep convolutional networks’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 1646–1654.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.5767
Loading

Related content

content/journals/10.1049/iet-ipr.2018.5767
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading