Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free Survey on GAN-based face hallucination with its model development

  • Author(s): Heng Liu 1, 2 ; Xiaoyu Zheng 1 ; Jungong Han 3 ; Yuezhong Chu 1 ; Tao Tao 1
    • View affiliations
  • Source: Volume 13, Issue 14, 12 December 2019, p. 2662 – 2672
    DOI:  10.1049/iet-ipr.2018.6545 , Print ISSN 1751-9659, Online ISSN 1751-9667
© The Institution of Engineering and Technology
Received 26/11/2018, Accepted 05/02/2019, Revised 21/01/2019, Published 15/02/2019

Face hallucination aims to produce a high-resolution face image from an input low-resolution face image, which is of great importance for many practical face applications, such as face recognition and face verification. Since the structure of the face image is complex and sensitive, obtaining a super-resolved face image is more difficult than generic image super-resolution. Recently, with great success in the high-level face recognition task, deep learning methods, especially generative adversarial networks (GANs), have also been applied to the low-level vision task – face hallucination. This work is to provide a model evolvement survey on GAN-based face hallucination. The principles of image resolution degradation and GAN-based learning are presented firstly. Then, a comprehensive review of the state-of-art GAN-based face hallucination methods is provided. Finally, the comparisons of these GAN-based face hallucination methods and the discussions of the related issues for future research direction are also provided.

Inspec keywords: face recognition; image resolution; learning (artificial intelligence)

Other keywords: GAN-based learning; state-of-art GAN-based face hallucination; practical face applications; face verification; image resolution degradation; input low-resolution face image; high-level face recognition task; generic image super-resolution; super-resolved face image; high-resolution face image

Subjects: Knowledge engineering techniques; Image recognition; Other topics in statistics; Optical, image and video signal processing; Other topics in statistics; Computer vision and image processing techniques

References

    1. 1)
      • 19. Dong, C., Loy, C. C., He, K., et al: ‘Image super-resolution using deep convolutional networks’, IEEE Trans. Pattern Anal. Mach. Intell., 2016, 38, (2), pp. 295–307.
    2. 2)
      • 9. Liu, C., Shum, H.-Y., Freeman, W. T.: ‘Face hallucination: theory and practice’, Int. J. Comput. Vis., 2007, 75, (1), pp. 115–134.
    3. 3)
      • 43. ‘Pytorch’. Available at https://github.com/pytorch/pytorch, accessed October 2018.
    4. 4)
      • 18. Bruna, J., Sprechmann, P., Lecun, Y.: ‘Super-resolution with deep convolutional sufficient statistics’, Comput. Sci., 2015, arXiv preprint arXiv:.05666, 2015.
    5. 5)
      • 42. Koestinger, M., Wohlhart, P., Roth, P. M., et al: ‘Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization’. 2011 IEEE Int. Conf. on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 2011, pp. 2144–2151.
    6. 6)
      • 31. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’, arXiv preprint arXiv:.1556, 2014.
    7. 7)
      • 6. Zheng, F., Tang, Y., Shao, L.: ‘Hetero-manifold regularization for cross-modal hashing’, IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40, (5), pp. 1059–1071.
    8. 8)
      • 11. Zhu, S., Liu, S., Loy, C. C., et al: ‘Deep cascaded bi-network for face hallucination’. European Conf. on Computer Vision, Amsterdam, Netherlands, 2016, pp. 614–630.
    9. 9)
      • 32. Bulat, A., Tzimiropoulos, G.: ‘Super-Fan: integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with Gans’, arXiv preprint arXiv:.02765, 2017.
    10. 10)
      • 25. Liu, Z., Luo, P., Wang, X., et al: ‘Deep learning face attributes in the wild’. IEEE Int. Conf. on Computer Vision, Santiago, Chile, 2015, pp. 3730–3738.
    11. 11)
      • 33. Bulat, A., Tzimiropoulos, G.: ‘How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)’. Int. Conf. on Computer Vision, Venice, Italy, 2017, p. 4.
    12. 12)
      • 21. Yu, X., Porikli, F.: ‘Ultra-resolving face images by discriminative generative networks’. European Conf. on Computer Vision, Amsterdam, Netherlands, 2016, pp. 318–333.
    13. 13)
      • 28. Ronneberger, O., Fischer, P., Brox, T.: ‘U-Net: convolutional networks for biomedical image segmentation’. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 2015, pp. 234–241.
    14. 14)
      • 34. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016, pp. 770–778.
    15. 15)
      • 10. Yang, C.-Y., Liu, S., Yang, M.-H.: ‘Structured face hallucination’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Portland, USA, 2013, pp. 1099–1106.
    16. 16)
      • 12. Dahl, R., Norouzi, M., Shlens, J.: ‘Pixel recursive super resolution’, arXiv preprint arXiv:.00783, 2017.
    17. 17)
      • 20. Kim, J., Lee, J. K., Lee, K. M.: ‘Accurate image super-resolution using very deep convolutional networks’. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016, pp. 1646–1654.
    18. 18)
      • 37. Chen, Y., Shen, C., Wei, X. S., et al: ‘Adversarial Posenet: a structure-aware convolutional network for human pose estimation’. Proc. IEEE Int. Conf. on Computer Vision, Venice, Italy, Oct 2017, pp. 1221–1230.
    19. 19)
      • 39. Collobert, R., Kavukcuoglu, K., Farabet, C.: ‘Torch7: an matlab-like environment for machine learning’. BigLearn, NIPS Workshop, Granada, Spain, 2011.
    20. 20)
      • 13. van den Oord, A., Kalchbrenner, N., Espeholt, L., et al: ‘Conditional image generation with Pixelcnn decoders’. Proc. Conf. and Workshop on Neural Information Processing Systems, Barcelona, Spain, Dec 2016, pp. 4797–4805.
    21. 21)
      • 41. Zhu, X., Lei, Z., Liu, X., et al: ‘Face alignment across large poses: a 3d solution’. Proc. of the IEEE Conf. on computer Vision and Pattern Recognition, Las Vegas, USA, 2016, pp. 146–155.
    22. 22)
      • 45. Wang, Z., Bovik, A.C., Sheikh, H.R., et al: ‘Image quality assessment: from error visibility to structural similarity’, IEEE Trans. Image Process., 2004, 13, (4), pp. 600–612.
    23. 23)
      • 3. Wang, Q., Wan, J., Yuan, Y.: ‘Deep metric learning for crowdedness regression’, IEEE Trans. Circuit Syst. Video Technol., 2018, 28, (10), pp. 2633–2643.
    24. 24)
      • 7. Baker, S., Kanade, T.: ‘Hallucinating faces’. Proc. Fourth IEEE Int. Conf. on Automatic Face and Gesture Recognition, Grenoble, France, 2000, pp. 83–88.
    25. 25)
      • 30. Yu, X., Fernando, B., Hartley, R., et al: ‘Super-resolving very low-resolution face images with supplementary attributes’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake, USA, 2018, pp. 908–917.
    26. 26)
      • 22. Chen, Z., Tong, Y.: ‘Face super-resolution through Wasserstein Gans’, arXiv preprint arXiv:.02438, 2017.
    27. 27)
      • 47. Tissera, M. D., McDonnell, M. D.: ‘Deep extreme learning machines: supervised autoencoding architecture for classification’, Neurocomputing, 2016, 174, pp. 42–49.
    28. 28)
      • 36. Newell, A., Yang, K., Deng, J.: ‘Stacked hourglass networks for human pose estimation’. European Conf. on Computer Vision, Amsterdam, Netherlands, 2016, pp. 483–499.
    29. 29)
      • 15. Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al: ‘Generative adversarial nets’. Proc. Conf. and Workshop on Neural Information Processing Systems, Montréal, Canada, Jul 2017, pp. 2672–2680.
    30. 30)
      • 38. Yu, X., Fernando, B., Ghanem, B., et al: ‘Face super-resolution guided by facial component heatmaps’. Proc. of the European Conf. on Computer Vision (ECCV), Munich, Germany, 2018, pp. 217–233.
    31. 31)
      • 16. Arjovsky, M., Chintala, S., Bottou, L.: ‘Wasserstein Gan’, arXiv preprint arXiv:.07875, 2017.
    32. 32)
      • 14. Ledig, C., Theis, L., Huszár, F., et al: ‘Photo-realistic single image super-resolution using a generative adversarial network’. Proc. Int. Conf. on Computer Vision and Pattern Recogintion, Hawaii, USA, Jul 2017, pp. 105–114.
    33. 33)
      • 35. Chen, Y., Tai, Y., Liu, X., et al: ‘Fsrnet: end-to-end learning face super-resolution with facial priors’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake, USA, 2018, pp. 2492–2501.
    34. 34)
      • 24. Radford, A., Metz, L., Chintala, S.: ‘Unsupervised representation learning with deep convolutional generative adversarial networks’, arXiv preprint arXiv:.06434, 2015.
    35. 35)
      • 1. Wang, Q., Chen, M., Nie, F., et al: ‘Detecting coherent groups in crowd scenes by multiview clustering’, IEEE Trans. Pattern Anal. Mach. Intell., 2018, to be published., DOI: 10.1109/TPAMI.2018.2875002.
    36. 36)
      • 26. Berthelot, D., Schumm, T., Metz, L.: ‘Began: boundary equilibrium generative adversarial networks’, arXiv preprint arXiv:.10717, 2017.
    37. 37)
      • 44. Le, V., Brandt, J., Lin, Z., et al: ‘Interactive facial feature localization’. European Conf. on Computer Vision, Florence, Italy, 2012, pp. 679–692.
    38. 38)
      • 4. Wang, Q., Qin, Z., Nie, F., et al: ‘Spectral embedded adaptive neighbors clustering’, IEEE Trans. Neural Netw. Learn. Syst., 2018, (99), pp. 1–7, to be published., DOI: 10.1109/TPAMI.2018.2875002.
    39. 39)
      • 5. Wang, Q., Wan, J., Nie, F., et al: ‘Hierarchical feature selection for random projection’, IEEE Trans. Neural Netw. Learn. Syst., 2018, to be published., DOI: 10.1109/TNNLS.2018.2868836.
    40. 40)
      • 29. Yu, X., Porikli, F.: ‘Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders’. Proc. Int. Conf. on Computer Vision and Pattern Recogintion, Hawaii, USA, Jul 2017, pp. 5367–5375.
    41. 41)
      • 40. Girija, S.S.: ‘Tensorflow: large-scale machine learning on heterogeneous distributed systems’, Software available from tensorflow.org, 2016.
    42. 42)
      • 23. Gulrajani, I., Ahmed, F., Arjovsky, M., et al: ‘Improved training of Wasserstein Gans’. Proc. Conf. and Workshop on Neural Information Processing Systems, Long Beach, USA, Jan 2017, pp. 5767–5777.
    43. 43)
      • 27. Huang, B., Chen, W., Wu, X., et al: ‘High-quality face image Sr using conditional generative adversarial networks’, arXiv preprint arXiv:.00737, 2017.
    44. 44)
      • 46. Wang, Y., Xie, Z., Xu, K., et al: ‘An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning’, Neurocomputing, 2016, 174, pp. 988–998.
    45. 45)
      • 17. Qi, G.-J.: ‘Loss-sensitive generative adversarial networks on Lipschitz densities’, arXiv preprint arXiv:.06264, 2017.
    46. 46)
      • 2. Zheng, F., Shao, L.: ‘A winner-take-All strategy for improved object tracking’, IEEE Trans. Image Process., 2018, 27, (9), pp. 4302–4313.
    47. 47)
      • 8. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution via sparse representation’, IEEE Trans. Image Process., 2010, 19, (11), pp. 2861–2873.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.6545
Loading

Related content

content/journals/10.1049/iet-ipr.2018.6545
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address