Advances in colour transfer
- Author(s): Francois Pitié 1
-
-
View affiliations
-
Affiliations:
1:
Electronic & Electrical Engineering Department, Trinity College Dublin , Ireland
-
Affiliations:
1:
Electronic & Electrical Engineering Department, Trinity College Dublin , Ireland
- Source:
Volume 14, Issue 6,
September
2020,
p.
304 – 322
DOI: 10.1049/iet-cvi.2019.0920 , Print ISSN 1751-9632, Online ISSN 1751-9640
Colour grading is an essential step in movie post-production, which is done in the industry by experienced artists on expensive edit hardware and software suites. This paper presents a review of the advances made to automate this process. The review looks in particular at how the state-of-the-art in optimal transport and deep learning has advanced some of the fundamental problems of colour transfer, and how far are we still from being able to automatically grade images.
Inspec keywords: video signal processing; art; image colour analysis; image registration; learning (artificial intelligence)
Other keywords: automatically grade images; colour grading; colour transfer; software suites; expensive edit hardware; optimal transport; movie post-production; experienced artists
Subjects: Video signal processing; Computer vision and image processing techniques; Optical, image and video signal processing; Knowledge engineering techniques
References
-
-
1)
-
23. Pele, O., Werman, M.: ‘Fast and robust earth mover's distances’. 2009 IEEE 12th Int. Conf. on Computer Vision, Kyoto, Japan, 2009, pp. 460–467.
-
-
2)
-
22. Rubner, Y., Tomasi, C., Guibas, L.J.: ‘The earth mover's distance as a metric for image retrieval’, Int. J. Comput. Vis., 2000, 40, (2), pp. 99–121.
-
-
3)
-
29. Bonneel, N., Rabin, J., Peyré, G., et al: ‘Sliced and radon wasserstein barycenters of measures’, J. Math. Imaging Vis., 2015, 51, (1), pp. 22–45.
-
-
4)
-
64. Li, Y., Wang, N., Liu, J., et al: ‘Demystifying neural style transfer’. Proc. of the 26th Int. Joint Conf. on Artificial Intelligence, IJCAI'17, Melbourne, Australia, 2017, pp. 2230–2236.
-
-
5)
-
43. Frigo, O., Sabater, N., Delon, J., et al: ‘Motion driven tonal stabilization’. 2015 IEEE Int. Conf. on Image Processing (ICIP), Québec city, Canada, 2015, pp. 3372–3376.
-
-
6)
-
6. Pitié, F., Kokaram, A., Dahyot, R.: ‘Automated colour grading using colour distribution transfer’, J. Comput. Vis. Image Understand., 2007, 107, (1–2), pp. 123–137.
-
-
7)
-
13. Pitié, F.: ‘Statistical signal processing techniques for visual post-production’. Ph.D. thesis, University of Dublin, Trinity College, 2006.
-
-
8)
-
57. Freedman, D., Kisilev, P.: ‘Object-to-object color transfer: optimal flows and SMSP transformations’. 2010 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 287–294.
-
-
9)
-
8. Pitié, F., Kokaram, A.: ‘The linear Monge-Kantorovitch linear colour mapping for example-based colour transfer’. 4th European Conf. on Visual Media Production, 2007. CVMP 2007, London, UK, 2007, pp. 1–9.
-
-
10)
-
45. Liu, J., Yang, W., Sun, X., et al: ‘Photo stylistic brush: robust style transfer via superpixel-based bipartite graph’, IEEE Trans. Multimed., 2018, 20, (7), pp. 1724–1737.
-
-
11)
-
42. Vazquez-Corral, J., Bertalmío, M.: ‘Color stabilization along time and across shots of the same scene, for one or several cameras of unknown specifications’, IEEE Trans. Image Process., 2014, 23, (10), pp. 4564–4575.
-
-
12)
-
32. Pérez, P., Blake, A., Gangnet, M.: ‘Poisson image editing’, ACM Trans. Graph. (SIGGRAPH'03), 2003, 22, (3), pp. 313–318.
-
-
13)
-
3. Morovic, J., Sun, P.-L.: ‘Accurate 3D image colour histogram transformation’, Pattern Recognit. Lett., 2003, 24, (11), pp. 1725–1735.
-
-
14)
-
60. Li, Y., Sharan, L., Adelson, E.H.: ‘Compressing and companding high dynamic range images with subband architectures’, ACM Trans. Graph., 2005, 24, (3), pp. 836–844.
-
-
15)
-
10. Abadpour, A., Kasaei, S.: ‘A fast and efficient fuzzy color transfer method’. Proc. of the IEEE Symp. on Signal Processing and Information Technology, Rome, Italy, 2004, pp. 491–494.
-
-
16)
-
21. Peyré, G., Cuturi, M.: ‘Computational optimal transport’. arXiv:1803.00567 [stat], arXiv: 1803.00567, 2018.
-
-
17)
-
44. Efros, A.A., Freeman, W.T.: ‘Image quilting for texture synthesis and transfer’. Proc. of the 28th Annual Conf. on Computer Graphics and Interactive Techniques, SIGGRAPH ‘01, New York, NY, USA, 2001, pp. 341–346.
-
-
18)
-
25. Wu, N., Coppins, R.: ‘Linear programming and extensions’ (McGraw-Hill, New York, 1981).
-
-
19)
-
14. Trussell, H.J., Vrhel, M.J.: ‘Color correction using principle components’, vol. 1452 (SPIE, San Jose, California, USA, 1991), pp. 2–9.
-
-
20)
-
34. Gatys, L.A., Ecker, A.S., Bethge, M., et al: ‘Controlling perceptual factors in neural style transfer’. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016, pp. 3730–3738.
-
-
21)
-
41. Park, J., Tai, Y.-W., Sinha, S.N., et al: ‘Efficient and robust color consistency for community photo collections’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, 2016, pp. 430–438.
-
-
22)
-
5. Pitié, F., Kokaram, A., Dahyot, R.: ‘Towards automated colour grading’. 2nd IEE European Conf. on Visual Media Production (CVMP'05), London, UK, 2005b.
-
-
23)
-
17. Evans, L.C.: ‘Partial differential equations and Monge-Kantorovich mass transfer’, Current Dev. Math., 1998, 1997, pp. 65–126.
-
-
24)
-
61. Sunkavalli, K., Johnson, M.K., Matusik, W., et al: ‘Multi-scale image harmonization’, ACM Trans. Graph., 2010, 29, (4), pp. 125:1–125:10.
-
-
25)
-
26. Cuturi, M.: ‘Sinkhorn distances: lightspeed computation of optimal transport’. Proc. of the 26th Int. Conf. on Neural Information Processing Systems – Volume 2, NIPS'13, Lake Tahoe, USA, 2013, pp. 2292–2300.
-
-
26)
-
30. Rabin, J., Peyré, G., Delon, J., et al: ‘Wasserstein barycenter and its application to texture mixing’. Scale Space and Variational Methods in Computer Vision, Ein-Gedi, Israel, 2012 (LNCS), pp. 435–446.
-
-
27)
-
55. Papadakis, N., Provenzi, E., Caselles, V.: ‘A variational model for histogram transfer of color images’, IEEE Trans. Image Process., 2011, 20, (6), pp. 1682–1695.
-
-
28)
-
15. Gonzalez, R.C., Woods, R.E.: ‘Digital image processing’ (Addison Wesley, Boston, USA, 1992).
-
-
29)
-
50. Silverman, B.W.: ‘Density estimation for statistics and data analysis’ (Chapman and Hall, Boca Raton, FL, USA, 1986).
-
-
30)
-
38. Shih, Y.-C., Paris, S., Durand, F., et al: ‘Data-driven hallucination of different times of day from a single outdoor photo’, ACM Trans. Graph., 2013, 32, pp. 200:1–200:11.
-
-
31)
-
46. Chen, L., Papandreou, G., Kokkinos, I., et al: ‘Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs’, IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40, (4), pp. 834–848.
-
-
32)
-
51. Jeong, K., Jaynes, C.: ‘Object matching in disjoint cameras using a color transfer approach’, Mach. Vis. Appl., 2008, 19, (5–6), pp. 443–455.
-
-
33)
-
2. Neumann, L., Neumann, A.: ‘Color style transfer techniques using hue, lightness and saturation histogram matching’. Computational Aesthetics, Girona, Spain, 2005a, pp. 111–122.
-
-
34)
-
48. Grogan, M., Dahyot, R.: ‘L2 divergence for robust colour transfer’, Comput. Vis. Image Underst., 2019, 181, pp. 39–49.
-
-
35)
-
28. Brenier, Y.: ‘Polar factorization and monotone rearrangement of vector-valued functions’, Commun. Pure Appl. Math., 1991, 44, (4), pp. 375–417.
-
-
36)
-
33. Pitié, F.: ‘An alternative matting Laplacian’. 2016 IEEE Int. Conf. on Image Processing (ICIP), Phoenix, Arizon, USA, 2016, pp. 3623–3627.
-
-
37)
-
12. Kotera, H.: ‘A scene-referred color transfer for pleasant imaging on display’. Proc. of the IEEE Int. Conf. on Image Processing, Genoa, Italy, 2005, pp. 5–8.
-
-
38)
-
49. Grogan, M., Dahyot, R., Smolic, A.: ‘User interaction for image recolouring using £2’. Proc. of the 14th European Conf. on Visual Media Production (CVMP 2017), London, UK, 2017, p. 6.
-
-
39)
-
20. Villani, C.: ‘Optimal transport: old and new’, vol. 338 (Springer Verlag, Berlin-Heidelberg-New York-Tokyo, 2009), https://doi.org/10.1007/978-3-540-71050-9.
-
-
40)
-
31. Levin, A., Lischinski, D., Weiss, Y.: ‘A closed-form solution to natural image matting’, IEEE Trans. Pattern Anal. Mach. Intell., 2008, 30, pp. 228–242.
-
-
41)
-
4. Pitié, F., Kokaram, A., Dahyot, R.: ‘N-dimensional probability density function transfer and its application to colour transfer’. Int. Conf. on Computer Vision (ICCV'05), Beijing, China, 2005a.
-
-
42)
-
53. Xu, S., Zhang, Y., Zhang, S., et al: ‘Uniform color transfer’. IEEE Int. Conf. on Image Processing 2005, Québec city, Canada, 2005, vol. 3, p. II–940.
-
-
43)
-
58. Ferradans, S., Papadakis, N., Peyré, G., et al: ‘Regularized discrete optimal transport’, SIAM J. Imag. Sci., 2014, 7, (3), pp. 1853–1882.
-
-
44)
-
16. Grundland, M., Dodgson, N.A.: ‘Color histogram specification by histogram warping’. Proc. of the SPIE Color Imaging X: Processing, Hardcopy, and Applications, San Jose, California, USA, 2005, vol. 5667, pp. 610–624.
-
-
45)
-
47. Grogan, M., Dahyot, R.: ‘Robust registration of Gaussian mixtures for colour transfer’. arXiv:1705.06091 [cs], arXiv: 1705.06091, 2017.
-
-
46)
-
9. Hwang, Y., Lee, J.-Y., So Kweon, I., et al: ‘Color transfer using probabilistic moving least squares’. Proc. of the IEEE Conf. on computer vision and pattern recognition, Columbus, OH, USA, 2014, pp. 3342–3349.
-
-
47)
-
35. Li, Y., Liu, M.-Y., Li, X., et al: ‘A closed-form solution to photorealistic image stylization’. ECCV, Munich, Germany, 2018.
-
-
48)
-
18. Gangbo, W., McCann, R.: ‘The geometry of optimal transport’, Acta Mathematica, 1996, 177, pp. 113–161.
-
-
49)
-
62. Gatys, L.A., Ecker, A.S., Bethge, M.: ‘Image style transfer using convolutional neural networks’. 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 2016, pp. 2414–2423.
-
-
50)
-
11. Abadpour, A., Kasaei, S.: ‘An efficient PCA-based color transfer method’, J. Vis. Commun. Image Represent., 2007, 18, (1), pp. 15–34.
-
-
51)
-
52. Xiang, Y., Zou, B., Li, H.: ‘Selective color transfer with multi-source images’, Pattern Recognit. Lett., 2009, 30, (7), pp. 682–689.
-
-
52)
-
19. Villani, C.: ‘Topics in optimal transportation’, vol. 58 of Graduate Studies in Mathematics (American Mathematical Society, Providence, RI, 2003).
-
-
53)
-
39. Faridul, H.S., Pouli, T., Chamaret, C., et al: ‘A survey of color mapping and its applications’, Eurographics (State of the Art Reports), 2014, 3, (see http://dx.doi.org/10.2312/egst.20141035).
-
-
54)
-
7. Pitie, F., Kokaram, A., Dahyot, R.: ‘Enhancement of digital photographs using color transfer techniques’, Image Processing Series. (CRC Press, Boca Raton, FL, USA, 2008), pp. 295–321.
-
-
55)
-
1. Reinhard, E., Stark, M., Shirley, P., et al: ‘Photographic tone reproduction for digital images’, ACM Trans. Graph. (Proc. ACM SIGGRAPH 2002), 2002, 21, (3), pp. 267–276.
-
-
56)
-
63. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’. Int. Conf. on Learning Representations, San Diego, CA, USA, 2015.
-
-
57)
-
24. Hitchcock, F.L.: ‘The distribution of a product from several sources to numerous localities’, J. Math. Phys., 1941, 20, pp. 224–230.
-
-
58)
-
56. Papadakis, N.: ‘Optimal transport for image processing’. Habilitation thesis, Habilitation à diriger des recherches, Université de Bordeaux, 2015.
-
-
59)
-
40. Oliveira, M., Sappa, A.D., Santos, V.: ‘A probabilistic approach for color correction in image mosaicking applications’, IEEE Trans. Image Process., 2014, 24, (2), pp. 508–523.
-
-
60)
-
37. Penhouet, S., Sanzenbacher, P.: ‘Automated deep photo style transfer’. arXiv:1901.03915 [cs], arXiv: 1901.03915, 2019.
-
-
61)
-
54. Neumann, L., Neumann, A.: ‘Color style transfer techniques using hue, lightness and saturation histogram matching’. Proc. of Computational Aestetics in Graphics, Visualization and Imaging, Girona, Spain, 2005b, pp. 111–122.
-
-
62)
-
27. Olkin, I., Pukelsheim, F.: ‘The distance between two random vectors with given dispersion matrices’, Linear Algebr. Appl., 1982, 48, pp. 257–263.
-
-
63)
-
36. Luan, F., Paris, S., Shechtman, E., et al: ‘Deep photo style transfer’. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Amsterdam, the Netherlands, 2017, pp. 6997–7005.
-
-
64)
-
59. Bae, S., Paris, S., Durand, F.: ‘Two-scale tone management for photographic look’, ACM Trans. Graph. (Proc. ACM SIGGRAPH 2006), 2006, 25, (3), pp. 637–645.
-
-
1)