access icon free Adaptive appearance separation for interactive image segmentation based on Dense CRF

© The Institution of Engineering and Technology
Received 25/02/2018, Accepted 22/10/2018, Revised 29/09/2018, Published 25/10/2018

Interactive segmentation has recently become a hot topic for its wide application. The authors propose an efficacious appearance separation model for interactive binary segmentation, which incorporates the difference of foreground and background colour models and the difference of corresponding geodesic models into the popular densely connected conditional random field (Dense CRF) framework. The proposed method can adaptively set relevant parameter values in this framework according to the characteristics of target images in a per-image manner, therefore, it gets rid of the dependence on specific datasets. After accomplishing a mean-field inference, the authors are able to get satisfactory results without the time-consuming parameter learning process and multiple iterative optimisations. Overall, the proposed approach is highly efficient and mitigates the contradiction between accuracy and segmentation efficiency. In addition, the proposed approach reduces the efforts of scribble-style interaction from users. The experimental results on three famous datasets show that the proposed method is superior to the other five new algorithms released in recent years regarding accuracy, and is faster than or close to them in runtime.

Inspec keywords: image segmentation; image colour analysis; learning (artificial intelligence)

Other keywords: interactive segmentation; background colour models; relevant parameter values; interactive image segmentation; popular densely connected conditional random field framework; Dense CRF; wide application; per-image manner; target images; segmentation efficiency; efficacious appearance separation model; corresponding geodesic models; hot topic; interactive binary segmentation; scribble-style interaction; time-consuming parameter; mean-field inference; foreground; adaptive appearance separation

Subjects: Optical, image and video signal processing; Computer vision and image processing techniques

References

    1. 1)
      • 49. Wang, W., Shen, J., Li, X., et al: ‘Robust video object cosegmentation’, IEEE Trans. Image Process., 2015, 24, (10), pp. 3137–3148.
    2. 2)
      • 17. Boykov, Y., Kolmogorov, V.: ‘An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision’, IEEE Trans. Pattern Anal Mach Intell., 2004, 26, (9), pp. 1124–1137.
    3. 3)
      • 59. Calinon, S., Guenter, F., Billard, A.: ‘On learning, representing, and generalizing a task in a humanoid robot’, IEEE Trans. Syst., Man, Cybern., B, 2007, 37, (2), pp. 286–298.
    4. 4)
      • 55. Achanta, R., Hemami, S.S., Estrada, F.J., et al: ‘Frequency-tuned salient region detection’. IEEE Conf. on Computer Vision and Pattern Recognition, IEEE Computer Society, Miami, Florida, USA, 2009, pp. 1597–1604.
    5. 5)
      • 28. Achanta, R., Shaji, A., Smith, K., et al: ‘SLIC superpixels compared to state-of-the-art superpixel methods’, IEEE Trans. Pattern Anal. Mach. Intell., 2012, 34, (11), pp. 2274–2282.
    6. 6)
      • 1. Li, Y., Sun, J., Tang, C., et al: ‘Lazy snapping’, ACM Trans. Graph., 2004, 23, (3), pp. 303–308.
    7. 7)
      • 62. Yatziv, L., Bartesaghi, A., Sapiro, G.: ‘O(N) implementation of the fast marching algorithm’, J. Comput. Phys., 2006, 212, (2), pp. 393–399.
    8. 8)
      • 21. Dong, X., Shen, J., Shao, L., et al: ‘Sub-markov random walk for image segmentation’, IEEE Trans. Image Process., 2016, 25, (2), pp. 516–527.
    9. 9)
      • 27. Comaniciu, D., Meer, P.: ‘Mean shift: a robust approach toward feature space analysis’, IEEE Trans. Pattern Anal. Mach. Intell., 2002, 24, (5), pp. 603–619.
    10. 10)
      • 25. Rezvanifar, A., Khosravifard, M.: ‘Including the size of regions in image segmentation by region-based graph’, IEEE Trans. Image Process., 2014, 23, (2), pp. 635–644.
    11. 11)
      • 26. Cheng, Y.: ‘Mean shift, mode seeking, and clustering’, IEEE Trans. Pattern Anal. Mach. Intell., 1995, 17, (8), pp. 790–799.
    12. 12)
      • 53. Arbelaez, P., Maire, M., Fowlkes, C.C., et al: ‘Contour detection and hierarchical image segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (5), pp. 898–916.
    13. 13)
      • 11. Vineet, V., Warrell, J., Torr, P.H.S.: ‘Filter-based mean-field inference for random fields with higher-order terms and product label-spaces’, Int. J. Comput. Vis., 2014, 110, (3), pp. 290–307.
    14. 14)
      • 15. Tang, M., Gorelick, L., Veksler, O., et al: ‘Grabcut in one cut’. IEEE Int. Conf. Computer Vision, 2013. IEEE Computer Society, Sydney, Australia, 2013, pp. 1769–1776.
    15. 15)
      • 41. Yang, X., Gao, X., Tao, D., et al: ‘An efficient MRF embedded level set method for image segmentation’, IEEE Trans. Image Process., 2015, 24, (1), pp. 9–21.
    16. 16)
      • 12. Desmaison, A., Bunel, R., Kohli, P., et al: ‘Efficient continuous relaxations for dense CRF’. European Conf. on Computer Vision, Amsterdam, The Netherlands, 2016, Proc., Part II (LNCS, 9906), 2016, pp. 818–833.
    17. 17)
      • 9. Adams, A., Baek, J., Davis, M.A.: ‘Fast high-dimensional filtering using the permutohedral lattice’, Comput. Graph. Forum, 2010, 29, (2), pp. 753–762.
    18. 18)
      • 6. Ning, J., Zhang, L., Zhang, D., et al: ‘Interactive image segmentation by maximal similarity based region merging’, Pattern Recognit., 2010, 43, (2), pp. 445–456.
    19. 19)
      • 31. Shen, J., Du, Y., Li, X.: ‘Interactive segmentation using constrained laplacian optimization’, IEEE Trans. Circuits Syst. Video Techn., 2014, 24, (7), pp. 1088–1100.
    20. 20)
      • 36. Krähenbühl, P., Koltun, V.: ‘Geodesic object proposals’. European Conf. on Computer Vision2014, Proc., Part V., Zurich, Switzerland (LNCS, 8693), 2014, pp. 725–739.
    21. 21)
      • 4. Boykov, Y., Jolly, M.: ‘Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images’. IEEE Int. Conf. on Computer Vision, Vancouver, British Columbia, Canada, 2001, pp. 105–112.
    22. 22)
      • 18. Cheng, M., Prisacariu, V.A., Zheng, S., et al: ‘Densecut: densely connected crfs for realtime grabcut’, Comput. Graph. Forum, 2015, 34, (7), pp. 193–201.
    23. 23)
      • 23. Shi, J., Malik, J.: ‘Normalized cuts and image segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2000, 22, (8), pp. 888–905.
    24. 24)
      • 52. Liu, T., Yuan, Z., Sun, J., et al: ‘Learning to detect a salient object’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (2), pp. 353–367.
    25. 25)
      • 56. Gorelick, L., Schmidt, F.R., Boykov, Y.: ‘Fast trust region for segmentation’. IEEE Conf. on Computer Vision and Pattern Recognition, 2013. IEEE Computer Society, Portland, OR, USA, 2013, pp. 1714–1721.
    26. 26)
      • 43. Cheng, M., Mitra, N.J., Huang, X., et al: ‘Global contrast based salient region detection’, IEEE Trans. Pattern Anal. Mach. Intell., 2015, 37, (3), pp. 569–582.
    27. 27)
      • 51. Orchard, M.T., Bouman, C.A.: ‘Color quantization of images’, IEEE Trans. Signal Process., 1991, 39, (12), pp. 2677–2690.
    28. 28)
      • 8. Krähenbühl, P., Koltun, V.: ‘Parameter learning and convergent inference for dense random fields’. Proc. of the 30th Int. Conf. on Machine Learning, 2013. vol. 28 of JMLR Workshop and Conference Proceedings. JMLR.org, Atlanta, GA, USA, 2013, pp. 513–521.
    29. 29)
      • 16. Rother, C., Kolmogorov, V., Blake, A.: ‘’Grabcut’: interactive foreground extraction using iterated graph cuts’, ACM Trans. Graph., 2004, 23, (3), pp. 309–314.
    30. 30)
      • 14. Gorelick, L., Veksler, O., Boykov, Y., et al: ‘Convexity shape prior for binary segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, (2), pp. 258–271.
    31. 31)
      • 22. Couprie, C., Grady, L.J., Najman, L., et al: ‘Power watershed: a unifying graph-based optimization framework’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (7), pp. 1384–1399.
    32. 32)
      • 29. Chew, S.E., Cahill, N.D.: ‘Semi-supervised normalized cuts for image segmentation’. IEEE Int. Conf. on Computer Vision, 2015. IEEE Computer Society, Santiago, Chile, 2015, pp. 1716–1723.
    33. 33)
      • 5. Blake, A., Rother, C., Brown, M.A., et al: ‘Interactive image segmentation using an adaptive GMMRF model’. 8th European Conf. Computer Vision, Prague, Czech Republic, 2004, (LNCS, 3021), pp. 428–441.
    34. 34)
      • 3. Boykov, Y., Jolly, M.: ‘Interactive organ segmentation using graph cuts’. Medical Image Computing and Computer-Assisted Intervention, Third Int. Conf., Pittsburgh, Pennsylvania, USA, 2000 (LNCS, 1935), pp. 276–286.
    35. 35)
      • 58. Calinon, S.: ‘Robot programming by demonstration - a probabilistic Approach’ (EPFL Press, 2009).
    36. 36)
      • 48. Han, J., Quan, R., Zhang, D., et al: ‘Robust object co-segmentation using background prior’, IEEE Trans. Image Process., 2018, 27, (4), pp. 1639–1651.
    37. 37)
      • 13. Ajanthan, T., Desmaison, A., Bunel, R., et al: ‘Efficient linear programming for dense crfs’. IEEE Conf. on Computer Vision and Pattern Recognition, 2017. IEEE Computer Society, Honolulu, HI, USA, 2017, pp. 2934–2942.
    38. 38)
      • 39. Wang, J., Yagi, Y.: ‘Shape prior embedded geodesic distance transform for image segmentation’. Asian Conf. on Computer Vision, 2010, Part II., Queenstown, New Zealand, (LNCS, 6469), 2010, pp. 72–81.
    39. 39)
      • 30. Ghanem, B., Ahuja, N.: ‘Dinkelbach NCUT: an efficient framework for solving normalized cuts problems with priors and convex constraints’, Int. J. Comput. Vis., 2010, 89, (1), pp. 40–55.
    40. 40)
      • 10. Vineet, V., Warrell, J., Sturgess, P., et al: ‘Improved initialization and Gaussian mixture pairwise terms for dense random fields with mean-field inference’. British Machine Vision Conf., Surrey, UK, September, 2012, pp. 1–11.
    41. 41)
      • 45. Wang, W., Shen, J., Yang, R., et al: ‘Saliency-aware video object segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40, (1), pp. 20–33.
    42. 42)
      • 20. Grady, L.: ‘Random walks for image segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28, (11), pp. 1768–1783.
    43. 43)
      • 37. Wang, W., Shen, J., Porikli, F.: ‘Saliency-aware geodesic video object segmentation’. IEEE Conf. on Computer Vision and Pattern Recognition, 2015. IEEE Computer Society, Boston, MA, USA, 2015, pp. 3395–3402.
    44. 44)
      • 34. Gulshan, V., Rother, C., Criminisi, A., et al: ‘Geodesic star convexity for interactive image segmentation’. IEEE Conf. Computer Vision and Pattern Recognition, 2010. IEEE Computer Society, San Francisco, CA, USA, 2010, pp. 3129–3136.
    45. 45)
      • 7. Krähenbühl, P., Koltun, V.: ‘Efficient inference in fully connected crfs with gaussian edge potentials’. Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems, Granada, Spain, 2011, pp. 109–117.
    46. 46)
      • 44. Ye, L., Liu, Z., Li, L., et al: ‘Salient object segmentation via effective integration of saliency and objectness’, IEEE Trans. Multimedia, 2017, 19, (8), pp. 1742–1756.
    47. 47)
      • 57. Gorelick, L., Boykov, Y., Veksler, O., et al: ‘Submodularization for binary pairwise energies’. IEEE Conf. on Computer Vision and Pattern Recognition, 2014. IEEE Computer Society, Columbus, OH, USA, 2014, pp. 1154–1161.
    48. 48)
      • 38. Feng, J., Price, B.L., Cohen, S., et al: ‘Interactive segmentation on RGBD images via cue selection’. IEEE Conf. on Computer Vision and Pattern Recognition, 2016. IEEE Computer Society, Las Vegas, NV, USA, 2016, pp. 156–164.
    49. 49)
      • 47. Dong, X., Shen, J., Shao, L., et al: ‘Interactive cosegmentation using global and local energy optimization’, IEEE Trans. Image Process., 2015, 24, (11), pp. 3966–3977.
    50. 50)
      • 35. Price, B.L., Morse, B.S., Cohen, S.: ‘Geodesic graph cut for interactive image segmentation’. IEEE Conf. on Computer Vision and Pattern Recognition, 2010. IEEE Computer Society, San Francisco, CA, USA, 2010, pp. 3161–3168.
    51. 51)
      • 24. Maji, S., Vishnoi, N.K., Malik, J.: ‘Biased normalized cuts’. IEEE Conf. on Computer Vision and Pattern Recognition, 2011, IEEE Computer Society, Colorado Springs, CO, USA, 2011, pp. 2057–2064.
    52. 52)
      • 50. Bai, X., Sapiro, G.: ‘A geodesic framework for fast interactive image and video segmentation and matting’. IEEE Int. Conf. on Computer Vision, 2007. IEEE Computer Society, Rio de Janeiro, Brazil, 2007, pp. 1–8.
    53. 53)
      • 19. Shen, J., Du, Y., Wang, W., et al: ‘Lazy random walks for superpixel segmentation’, IEEE Trans. Image Process., 2014, 23, (4), pp. 1451–1462.
    54. 54)
      • 61. Toivanen, P.J.: ‘Erratum to ‘new geodesic distance transforms for gray-scale images’, Pattern Recognit. Lett., 1996, 17, (13), pp. 437–450.
    55. 55)
      • 42. Khadidos, A., Sanchez, V., Li, C.: ‘Weighted level set evolution based on local edge features for medical image segmentation’, IEEE Trans. Image Process., 2017, 26, (4), pp. 1979–1991.
    56. 56)
      • 32. Criminisi, A., Sharp, T., Blake, A.: ‘Geos: geodesic image segmentation’. European Conf. on Computer VisionProc., Part I., Marseille, France, 2008, (LNCS, 5302), pp. 99–112.
    57. 57)
      • 60. Yang, C., Duraiswami, R., Gumerov, N.A., et al: ‘Improved fast gauss transform and efficient kernel density estimation’. IEEE Int. Conf. Computer Vision, 2003, IEEE Computer Society, Nice, France, 2003, pp. 464–471.
    58. 58)
      • 33. Criminisi, A., Sharp, T., Rother, C., et al: ‘Geodesic image and video editing’, ACM Trans. Graph., 2010, 29, (5), pp. 134:1–134:15.
    59. 59)
      • 54. Dai, J., He, K., Sun, J.: ‘Instance-aware semantic segmentation via multi-task network cascades’. IEEE Conf. on Computer Vision and Pattern Recognition, 2016. IEEE Computer Society, Las Vegas, NV, USA, 2016, pp. 3150–3158.
    60. 60)
      • 40. Li, C., Xu, C., Gui, C., et al: ‘Distance regularized level set evolution and its application to image segmentation’, IEEE Trans. Image Process., 2010, 19, (12), pp. 3243–3254.
    61. 61)
      • 46. Syu, J., Wang, S., Wang, L.: ‘Hierarchical image segmentation based on iterative contraction and merging’, IEEE Trans. Image Process., 2017, 26, (5), pp. 2246–2260.
    62. 62)
      • 2. Li, C., Xu, C., Gui, C., et al: ‘Level set evolution without re-initialization: A new variational formulation’. IEEE Conf. Computer Vision and Pattern Recognition, San Diego, CA, USA, 2005, pp. 430–436.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.5073
Loading

Related content

content/journals/10.1049/iet-ipr.2018.5073
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading