access icon free Creative and diverse artwork generation using adversarial networks

Existing style transfer methods have achieved great success in artwork generation by transferring artistic styles onto everyday photographs while keeping their contents unchanged. Despite this success, these methods have one inherent limitation: they cannot produce newly created image contents, lacking creativity and flexibility. On the other hand, generative adversarial networks (GANs) can synthesise images with new content, whereas cannot specify the artistic style of these images. The authors consider combining style transfer with convolutional GANs to generate more creative and diverse artworks. Instead of simply concatenating these two networks: the first for synthesising new content and the second for transferring artistic styles, which is inefficient and inconvenient, they design an end-to-end network called ArtistGAN to perform these two operations at the same time and achieve visually better results. Moreover, to generate images of higher quality, they propose the bi-discriminator GAN containing a pixel discriminator and a feature discriminator that constrain the generated image from pixel level and feature level, respectively. They conduct extensive experiments and comparisons to evaluate their methods quantitatively and qualitatively. The experimental results verify the effectiveness of their methods.

Inspec keywords: photography; art; convolutional neural nets; unsupervised learning; image texture

Other keywords: photographs; creative artwork generation; artistic style; end-to-end network; feature discriminator; image generation; ArtistGAN; convolutional GAN; bidiscriminator GAN; convolutional generative adversarial networks; style transfer methods; diverse artwork generation; pixel discriminator

Subjects: Computer vision and image processing techniques; Neural computing techniques; Humanities computing; Optical, image and video signal processing

References

    1. 1)
      • 40. Salimans, T., Goodfellow, I., Zaremba, W., et al: ‘Improved techniques for training GANs’. Advances in Neural Information Processing Systems, Barcelona, Spain, December 2016, pp. 22342242.
    2. 2)
      • 8. Yao, Y., Ren, J., Xie, X., et al: ‘Attention-aware multi-stroke style transfer’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Long Beach, CA, USA, June 2019, pp. 14671475.
    3. 3)
      • 10. Huang, X., Belongie, S.: ‘Arbitrary style transfer in real-time with adaptive instance normalization’. Proc. IEEE Int. Conf. on Computer Vision, Venice, Italy, October 2017, pp. 15011510.
    4. 4)
      • 41. Heusel, M., Ramsauer, H., Unterthiner, T., et al: ‘GANs trained by a two time-scale update rule converge to a local Nash equilibrium’. Advances in Neural Information Processing Systems, Long Beach, CA, USA, December 2017, pp. 66266637.
    5. 5)
      • 19. Risser, E., Wilmot, P., Barnes, C.: ‘Stable and controllable neural texture synthesis and style transfer using histogram losses’, arXiv preprint arXiv:170108893, 2017.
    6. 6)
      • 42. Russakovsky, O., Deng, J., Su, H., et al: ‘ImageNet large scale visual recognition challenge’, Int. J. Comput. Vis., 2015, 115, (3), pp. 211252.
    7. 7)
      • 3. Li, C., Wand, M.: ‘Precomputed real-time texture synthesis with Markovian generative adversarial networks’. European Conf. on Computer Vision, Amsterdam, The Netherlands, October 2016, pp. 702716.
    8. 8)
      • 25. Zhu, J.Y., Park, T., Isola, P., et al: ‘Unpaired image-to-image translation using cycle-consistent adversarial networks’. Proc. IEEE Int. Conf. on Computer Vision, Venice, Italy, October 2017, pp. 22232232.
    9. 9)
      • 35. Mahendran, A., Vedaldi, A.: ‘Understanding deep image representations by inverting them’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Boston, MA, USA, June 2015, pp. 51885196.
    10. 10)
      • 17. Zhang, H., Goodfellow, I., Metaxas, D., et al: ‘Self-attention generative adversarial networks’, arXiv preprint arXiv:180508318, 2018.
    11. 11)
      • 20. Ledig, C., Theis, L., Huszár, F., et al: ‘Photo-realistic single image super-resolution using a generative adversarial network’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition., Honolulu, Hawaii, USA, July 2017, pp. 46814690.
    12. 12)
      • 2. Johnson, J., Alahi, A., Fei-Fei, L.: ‘Perceptual losses for real-time style transfer and super-resolution’. European Conf. on Computer Vision, Amsterdam, The Netherlands, October 2016, pp. 694711.
    13. 13)
      • 6. Ulyanov, D., Vedaldi, A., Lempitsky, V.: ‘Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, Hawaii, USA, July 2017, pp. 69246932.
    14. 14)
      • 7. Wang, X., Oxholm, G., Zhang, D., et al: ‘Multimodal transfer: a hierarchical deep convolutional neural network for fast artistic style transfer’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, Hawaii, USA, July 2017, pp. 52395247.
    15. 15)
      • 9. Dumoulin, V., Shlens, J., Kudlur, M.: ‘A learned representation for artistic style’, arXiv preprint arXiv:161007629, 2016.
    16. 16)
      • 39. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, June 2016, pp. 770778.
    17. 17)
      • 15. Radford, A., Metz, L., Chintala, S.: ‘Unsupervised representation learning with deep convolutional generative adversarial networks’, arXiv preprint arXiv:151106434, 2015.
    18. 18)
      • 33. Gulrajani, I., Ahmed, F., Arjovsky, M., et al: ‘Improved training of Wasserstein GANs’. Advances in Neural Information Processing Systems, Long Beach, CA, USA, December 2017, pp. 57675777.
    19. 19)
      • 21. Wang, X., Yu, K., Wu, S., et al: ‘ESRGAN: enhanced super-resolution generative adversarial networks’. Proc. European Conf. on Computer Vision (ECCV), Munich, Germany, September 2018.
    20. 20)
      • 4. Ulyanov, D., Lebedev, V., Vedaldi, A., et al: ‘Texture networks: feed-forward synthesis of textures and stylized images’. ICML, New York City, NY, USA, June 2016, vol. 1, p. 4.
    21. 21)
      • 11. Li, Y., Fang, C., Yang, J., et al: ‘Universal style transfer via feature transforms’. Advances in Neural Information Processing Systems, Long Beach, CA, USA, December 2017, pp. 386396.
    22. 22)
      • 14. Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al: ‘Generative adversarial nets’. Advances in Neural Information Processing Systems, Montreal, Quebec, Canada, December 2014, pp. 26722680.
    23. 23)
      • 5. Li, C., Wand, M.: ‘Combining Markov random fields and convolutional neural networks for image synthesis’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, June 2016, pp. 24792486.
    24. 24)
      • 30. Yu, J., Lin, Z., Yang, J., et al: ‘Generative image inpainting with contextual attention’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, Utah, USA, June 2018, pp. 55055514.
    25. 25)
      • 24. Isola, P., Zhu, J.Y., Zhou, T., et al: ‘Image-to-image translation with conditional adversarial networks’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, Hawaii, USA, July 2017, pp. 11251134.
    26. 26)
      • 22. Luc, P., Couprie, C., Chintala, S., et al: ‘Semantic segmentation using adversarial networks’, arXiv preprint arXiv:161108408, 2016.
    27. 27)
      • 43. Nichol, K.: ‘Painter by numbers, wikiart’, 2016. Available at https://www.kaggle.com/c/painter-by-numbers.
    28. 28)
      • 18. Brock, A., Donahue, J., Simonyan, K.: ‘Large scale GAN training for high fidelity natural image synthesis’, arXiv preprint arXiv:180911096, 2018.
    29. 29)
      • 36. Sungatullina, D., Zakharov, E., Ulyanov, D., et al: ‘Image manipulation with perceptual discriminators’. Proc. European Conf. on Computer Vision (ECCV), Munich, Germany, September 2018, pp. 579595.
    30. 30)
      • 34. Miyato, T., Kataoka, T., Koyama, M., et al: ‘Spectral normalization for generative adversarial networks’, arXiv preprint arXiv:180205957, 2018.
    31. 31)
      • 37. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’, arXiv preprint arXiv:14091556, 2014.
    32. 32)
      • 16. Arjovsky, M., Chintala, S., Bottou, L.: ‘Wasserstein GAN’, arXiv preprint arXiv:170107875, 2017.
    33. 33)
      • 23. Hung, W.C., Tsai, Y.H., Liou, Y.T., et al: ‘Adversarial learning for semi-supervised semantic segmentation’, arXiv preprint arXiv:180207934, 2018.
    34. 34)
      • 27. Huang, X., Liu, M.Y., Belongie, S., et al: ‘Multimodal unsupervised image-to-image translation’. Proc. European Conf. on Computer Vision (ECCV), Munich, Germany, September 2018, pp. 172189.
    35. 35)
      • 12. Ghiasi, G., Lee, H., Kudlur, M., et al: ‘Exploring the structure of a real-time, arbitrary neural artistic stylization network’, arXiv preprint arXiv:170506830, 2017.
    36. 36)
      • 28. Pathak, D., Krahenbuhl, P., Donahue, J., et al: ‘Context encoders: feature learning by inpainting’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, June 2016, pp. 25362544.
    37. 37)
      • 31. Mao, X., Li, Q., Xie, H., et al: ‘Least squares generative adversarial networks’. Proc. IEEE Int. Conf. on Computer Vision, Venice, Italy, October 2017, pp. 27942802.
    38. 38)
      • 32. Lim, J.H., Ye, J.C.: ‘Geometric GAN’, arXiv preprint arXiv:170502894, 2017.
    39. 39)
      • 26. Liu, M.Y., Breuel, T., Kautz, J.: ‘Unsupervised image-to-image translation networks’. Advances in Neural Information Processing Systems, Long Beach, CA, USA, December 2017, pp. 700708.
    40. 40)
      • 13. Sheng, L., Lin, Z., Shao, J., et al: ‘Avatar-Net: multi-scale zero-shot style transfer by feature decoration’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, Utah, USA, June 2018, pp. 82428250.
    41. 41)
      • 1. Gatys, L.A., Ecker, A.S., Bethge, M.: ‘Image style transfer using convolutional neural networks’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, June 2016, pp. 24142423.
    42. 42)
      • 38. Yu, F., Seff, A., Zhang, Y., et al: ‘LSUN: construction of a large-scale image dataset using deep learning with humans in the loop’, arXiv preprint arXiv:150603365, 2015.
    43. 43)
      • 29. Li, Y., Liu, S., Yang, J., et al: ‘Generative face completion’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, Hawaii, USA, July 2017, pp. 39113919.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2020.0014
Loading

Related content

content/journals/10.1049/iet-cvi.2020.0014
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading