Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon openaccess Advances in colour transfer

Colour grading is an essential step in movie post-production, which is done in the industry by experienced artists on expensive edit hardware and software suites. This paper presents a review of the advances made to automate this process. The review looks in particular at how the state-of-the-art in optimal transport and deep learning has advanced some of the fundamental problems of colour transfer, and how far are we still from being able to automatically grade images.

References

    1. 1)
      • 23. Pele, O., Werman, M.: ‘Fast and robust earth mover's distances’. 2009 IEEE 12th Int. Conf. on Computer Vision, Kyoto, Japan, 2009, pp. 460467.
    2. 2)
      • 22. Rubner, Y., Tomasi, C., Guibas, L.J.: ‘The earth mover's distance as a metric for image retrieval’, Int. J. Comput. Vis., 2000, 40, (2), pp. 99121.
    3. 3)
      • 29. Bonneel, N., Rabin, J., Peyré, G., et al: ‘Sliced and radon wasserstein barycenters of measures’, J. Math. Imaging Vis., 2015, 51, (1), pp. 2245.
    4. 4)
      • 64. Li, Y., Wang, N., Liu, J., et al: ‘Demystifying neural style transfer’. Proc. of the 26th Int. Joint Conf. on Artificial Intelligence, IJCAI'17, Melbourne, Australia, 2017, pp. 22302236.
    5. 5)
      • 43. Frigo, O., Sabater, N., Delon, J., et al: ‘Motion driven tonal stabilization’. 2015 IEEE Int. Conf. on Image Processing (ICIP), Québec city, Canada, 2015, pp. 33723376.
    6. 6)
      • 6. Pitié, F., Kokaram, A., Dahyot, R.: ‘Automated colour grading using colour distribution transfer’, J. Comput. Vis. Image Understand., 2007, 107, (1–2), pp. 123137.
    7. 7)
      • 13. Pitié, F.: ‘Statistical signal processing techniques for visual post-production’. Ph.D. thesis, University of Dublin, Trinity College, 2006.
    8. 8)
      • 57. Freedman, D., Kisilev, P.: ‘Object-to-object color transfer: optimal flows and SMSP transformations’. 2010 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 287294.
    9. 9)
      • 8. Pitié, F., Kokaram, A.: ‘The linear Monge-Kantorovitch linear colour mapping for example-based colour transfer’. 4th European Conf. on Visual Media Production, 2007. CVMP 2007, London, UK, 2007, pp. 19.
    10. 10)
      • 45. Liu, J., Yang, W., Sun, X., et al: ‘Photo stylistic brush: robust style transfer via superpixel-based bipartite graph’, IEEE Trans. Multimed., 2018, 20, (7), pp. 17241737.
    11. 11)
      • 42. Vazquez-Corral, J., Bertalmío, M.: ‘Color stabilization along time and across shots of the same scene, for one or several cameras of unknown specifications’, IEEE Trans. Image Process., 2014, 23, (10), pp. 45644575.
    12. 12)
      • 32. Pérez, P., Blake, A., Gangnet, M.: ‘Poisson image editing’, ACM Trans. Graph. (SIGGRAPH'03), 2003, 22, (3), pp. 313318.
    13. 13)
      • 3. Morovic, J., Sun, P.-L.: ‘Accurate 3D image colour histogram transformation’, Pattern Recognit. Lett., 2003, 24, (11), pp. 17251735.
    14. 14)
      • 60. Li, Y., Sharan, L., Adelson, E.H.: ‘Compressing and companding high dynamic range images with subband architectures’, ACM Trans. Graph., 2005, 24, (3), pp. 836844.
    15. 15)
      • 10. Abadpour, A., Kasaei, S.: ‘A fast and efficient fuzzy color transfer method’. Proc. of the IEEE Symp. on Signal Processing and Information Technology, Rome, Italy, 2004, pp. 491494.
    16. 16)
      • 21. Peyré, G., Cuturi, M.: ‘Computational optimal transport’. arXiv:1803.00567 [stat], arXiv: 1803.00567, 2018.
    17. 17)
      • 44. Efros, A.A., Freeman, W.T.: ‘Image quilting for texture synthesis and transfer’. Proc. of the 28th Annual Conf. on Computer Graphics and Interactive Techniques, SIGGRAPH ‘01, New York, NY, USA, 2001, pp. 341346.
    18. 18)
      • 25. Wu, N., Coppins, R.: ‘Linear programming and extensions’ (McGraw-Hill, New York, 1981).
    19. 19)
      • 14. Trussell, H.J., Vrhel, M.J.: ‘Color correction using principle components’, vol. 1452 (SPIE, San Jose, California, USA, 1991), pp. 29.
    20. 20)
      • 34. Gatys, L.A., Ecker, A.S., Bethge, M., et al: ‘Controlling perceptual factors in neural style transfer’. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016, pp. 37303738.
    21. 21)
      • 41. Park, J., Tai, Y.-W., Sinha, S.N., et al: ‘Efficient and robust color consistency for community photo collections’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, 2016, pp. 430438.
    22. 22)
      • 5. Pitié, F., Kokaram, A., Dahyot, R.: ‘Towards automated colour grading’. 2nd IEE European Conf. on Visual Media Production (CVMP'05), London, UK, 2005b.
    23. 23)
      • 17. Evans, L.C.: ‘Partial differential equations and Monge-Kantorovich mass transfer’, Current Dev. Math., 1998, 1997, pp. 65126.
    24. 24)
      • 61. Sunkavalli, K., Johnson, M.K., Matusik, W., et al: ‘Multi-scale image harmonization’, ACM Trans. Graph., 2010, 29, (4), pp. 125:1125:10.
    25. 25)
      • 26. Cuturi, M.: ‘Sinkhorn distances: lightspeed computation of optimal transport’. Proc. of the 26th Int. Conf. on Neural Information Processing Systems – Volume 2, NIPS'13, Lake Tahoe, USA, 2013, pp. 22922300.
    26. 26)
      • 30. Rabin, J., Peyré, G., Delon, J., et al: ‘Wasserstein barycenter and its application to texture mixing’. Scale Space and Variational Methods in Computer Vision, Ein-Gedi, Israel, 2012 (LNCS), pp. 435446.
    27. 27)
      • 55. Papadakis, N., Provenzi, E., Caselles, V.: ‘A variational model for histogram transfer of color images’, IEEE Trans. Image Process., 2011, 20, (6), pp. 16821695.
    28. 28)
      • 15. Gonzalez, R.C., Woods, R.E.: ‘Digital image processing’ (Addison Wesley, Boston, USA, 1992).
    29. 29)
      • 50. Silverman, B.W.: ‘Density estimation for statistics and data analysis’ (Chapman and Hall, Boca Raton, FL, USA, 1986).
    30. 30)
      • 38. Shih, Y.-C., Paris, S., Durand, F., et al: ‘Data-driven hallucination of different times of day from a single outdoor photo’, ACM Trans. Graph., 2013, 32, pp. 200:1200:11.
    31. 31)
      • 46. Chen, L., Papandreou, G., Kokkinos, I., et al: ‘Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs’, IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40, (4), pp. 834848.
    32. 32)
      • 51. Jeong, K., Jaynes, C.: ‘Object matching in disjoint cameras using a color transfer approach’, Mach. Vis. Appl., 2008, 19, (5–6), pp. 443455.
    33. 33)
      • 2. Neumann, L., Neumann, A.: ‘Color style transfer techniques using hue, lightness and saturation histogram matching’. Computational Aesthetics, Girona, Spain, 2005a, pp. 111122.
    34. 34)
      • 48. Grogan, M., Dahyot, R.: ‘L2 divergence for robust colour transfer’, Comput. Vis. Image Underst., 2019, 181, pp. 3949.
    35. 35)
      • 28. Brenier, Y.: ‘Polar factorization and monotone rearrangement of vector-valued functions’, Commun. Pure Appl. Math., 1991, 44, (4), pp. 375417.
    36. 36)
      • 33. Pitié, F.: ‘An alternative matting Laplacian’. 2016 IEEE Int. Conf. on Image Processing (ICIP), Phoenix, Arizon, USA, 2016, pp. 36233627.
    37. 37)
      • 12. Kotera, H.: ‘A scene-referred color transfer for pleasant imaging on display’. Proc. of the IEEE Int. Conf. on Image Processing, Genoa, Italy, 2005, pp. 58.
    38. 38)
      • 49. Grogan, M., Dahyot, R., Smolic, A.: ‘User interaction for image recolouring using £2’. Proc. of the 14th European Conf. on Visual Media Production (CVMP 2017), London, UK, 2017, p. 6.
    39. 39)
      • 20. Villani, C.: ‘Optimal transport: old and new’, vol. 338 (Springer Verlag, Berlin-Heidelberg-New York-Tokyo, 2009), https://doi.org/10.1007/978-3-540-71050-9.
    40. 40)
      • 31. Levin, A., Lischinski, D., Weiss, Y.: ‘A closed-form solution to natural image matting’, IEEE Trans. Pattern Anal. Mach. Intell., 2008, 30, pp. 228242.
    41. 41)
      • 4. Pitié, F., Kokaram, A., Dahyot, R.: ‘N-dimensional probability density function transfer and its application to colour transfer’. Int. Conf. on Computer Vision (ICCV'05), Beijing, China, 2005a.
    42. 42)
    43. 43)
      • 58. Ferradans, S., Papadakis, N., Peyré, G., et al: ‘Regularized discrete optimal transport’, SIAM J. Imag. Sci., 2014, 7, (3), pp. 18531882.
    44. 44)
      • 16. Grundland, M., Dodgson, N.A.: ‘Color histogram specification by histogram warping’. Proc. of the SPIE Color Imaging X: Processing, Hardcopy, and Applications, San Jose, California, USA, 2005, vol. 5667, pp. 610624.
    45. 45)
      • 47. Grogan, M., Dahyot, R.: ‘Robust registration of Gaussian mixtures for colour transfer’. arXiv:1705.06091 [cs], arXiv: 1705.06091, 2017.
    46. 46)
      • 9. Hwang, Y., Lee, J.-Y., So Kweon, I., et al: ‘Color transfer using probabilistic moving least squares’. Proc. of the IEEE Conf. on computer vision and pattern recognition, Columbus, OH, USA, 2014, pp. 33423349.
    47. 47)
      • 35. Li, Y., Liu, M.-Y., Li, X., et al: ‘A closed-form solution to photorealistic image stylization’. ECCV, Munich, Germany, 2018.
    48. 48)
      • 18. Gangbo, W., McCann, R.: ‘The geometry of optimal transport’, Acta Mathematica, 1996, 177, pp. 113161.
    49. 49)
      • 62. Gatys, L.A., Ecker, A.S., Bethge, M.: ‘Image style transfer using convolutional neural networks’. 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016, pp. 24142423.
    50. 50)
      • 11. Abadpour, A., Kasaei, S.: ‘An efficient PCA-based color transfer method’, J. Vis. Commun. Image Represent., 2007, 18, (1), pp. 1534.
    51. 51)
      • 52. Xiang, Y., Zou, B., Li, H.: ‘Selective color transfer with multi-source images’, Pattern Recognit. Lett., 2009, 30, (7), pp. 682689.
    52. 52)
      • 19. Villani, C.: ‘Topics in optimal transportation’, vol. 58 of Graduate Studies in Mathematics (American Mathematical Society, Providence, RI, 2003).
    53. 53)
      • 39. Faridul, H.S., Pouli, T., Chamaret, C., et al: ‘A survey of color mapping and its applications’, Eurographics (State of the Art Reports), 2014, 3, (see http://dx.doi.org/10.2312/egst.20141035).
    54. 54)
      • 7. Pitie, F., Kokaram, A., Dahyot, R.: ‘Enhancement of digital photographs using color transfer techniques’, Image Processing Series. (CRC Press, Boca Raton, FL, USA, 2008), pp. 295321.
    55. 55)
      • 1. Reinhard, E., Stark, M., Shirley, P., et al: ‘Photographic tone reproduction for digital images’, ACM Trans. Graph. (Proc. ACM SIGGRAPH 2002), 2002, 21, (3), pp. 267276.
    56. 56)
      • 63. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’. Int. Conf. on Learning Representations, San Diego, CA, USA, 2015.
    57. 57)
      • 24. Hitchcock, F.L.: ‘The distribution of a product from several sources to numerous localities’, J. Math. Phys., 1941, 20, pp. 224230.
    58. 58)
      • 56. Papadakis, N.: ‘Optimal transport for image processing’. Habilitation thesis, Habilitation à diriger des recherches, Université de Bordeaux, 2015.
    59. 59)
      • 40. Oliveira, M., Sappa, A.D., Santos, V.: ‘A probabilistic approach for color correction in image mosaicking applications’, IEEE Trans. Image Process., 2014, 24, (2), pp. 508523.
    60. 60)
      • 37. Penhouet, S., Sanzenbacher, P.: ‘Automated deep photo style transfer’. arXiv:1901.03915 [cs], arXiv: 1901.03915, 2019.
    61. 61)
      • 54. Neumann, L., Neumann, A.: ‘Color style transfer techniques using hue, lightness and saturation histogram matching’. Proc. of Computational Aestetics in Graphics, Visualization and Imaging, Girona, Spain, 2005b, pp. 111122.
    62. 62)
      • 27. Olkin, I., Pukelsheim, F.: ‘The distance between two random vectors with given dispersion matrices’, Linear Algebr. Appl., 1982, 48, pp. 257263.
    63. 63)
      • 36. Luan, F., Paris, S., Shechtman, E., et al: ‘Deep photo style transfer’. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Amsterdam, the Netherlands, 2017, pp. 69977005.
    64. 64)
      • 59. Bae, S., Paris, S., Durand, F.: ‘Two-scale tone management for photographic look’, ACM Trans. Graph. (Proc. ACM SIGGRAPH 2006), 2006, 25, (3), pp. 637645.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2019.0920
Loading

Related content

content/journals/10.1049/iet-cvi.2019.0920
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address