Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free EasyFlow: increasing the convergence basin of variational image matching with a feature-based cost

© The Institution of Engineering and Technology
Received 22/04/2016, Accepted 06/09/2016, Revised 29/08/2016, Published 16/09/2016

Dense motion field estimation is a key computer vision problem. Many solutions have been proposed to compute small or large displacements, narrow or wide baseline stereo disparity, or non-rigid surface registration, but a unified methodology is still lacking. The authors introduce a general framework that robustly combines direct and feature-based matching. The feature-based cost is built around a novel robust distance function that handles keypoints and weak features such as segments. It allows us to use putative feature matches to guide dense motion estimation out of local minima. The authors’ framework uses a robust direct data term. It is implemented with a powerful second-order regularisation with external and self-occlusion reasoning. Their framework achieves state-of-the-art performance in several cases (standard optical flow benchmarks, wide-baseline stereo and non-rigid surface registration). Their framework has a modular design that customises to specific application needs.

Inspec keywords: image matching; image sequences; computer vision

Other keywords: convergence basin; second-order regularisation; surface registration; optical flow; self-occlusion reasoning; computer vision; variational image matching; feature-based matching; dense motion field estimation; EasyFlow; feature-based cost; baseline stereo disparity

Subjects: Computer vision and image processing techniques; Image recognition

References

    1. 1)
      • 18. Snavely, N., Seitz, S.M., Szeliski, R.: ‘Photo tourism: exploring photo collections in 3d’, ACM Trans. Graph. (Proc. SIGGRAPH), 2006, 25, (3), pp. 835–846.
    2. 2)
      • 32. Fischler, M.A., Bolles, R.C.: ‘Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography’, Commun. ACM, 1981, 24, (6), pp. 381–395.
    3. 3)
      • 52. Barnes, C., Shechtman, E., Finkelstein, A., et al: ‘PatchMatch: a randomized correspondence algorithm for structural image editing’, ACM Trans. Graph. (Proc. SIGGRAPH), 2009, 28, (3).
    4. 4)
      • 11. Bay, H., Tuytelaars, T., Van Gool, L.: ‘SURF: speeded up robust features’. European Conf. on Computer Vision (ECCV), 2006.
    5. 5)
      • 6. Bredies, K., Kunisch, K., Pock, T.: ‘Total generalized variation’, SIAM J. Imaging Sci., 2010, 3, (3), pp. 492–526.
    6. 6)
      • 21. Gay-Bellile, V., Bartoli, A., Sayd, P.: ‘Direct estimation of nonrigid registrations with image-based self-occlusion reasoning’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (1), pp. 87–104.
    7. 7)
      • 4. Werlberger, M., Trobin, W., Pock, T., et al: ‘Anisotropic Huber-L1 optical flow’. British Machine Vision Conf. (BMVC), 2009.
    8. 8)
      • 39. Baker, S., Scharstein, D., Lewis, J., et al: ‘A database and evaluation methodology for optical flow’, Int. J. Comput. Vis., 2011, 92, (1), pp. 1–31.
    9. 9)
      • 43. Revaud, J., Weinzaepfel, P., Harchaoui, Z., et al: ‘EpicFlow: edge-preserving interpolation of correspondences for optical flow’. CVPR 2015 – IEEE Conf. on Computer Vision & Pattern Recognition, Boston, United States, June 2015.
    10. 10)
      • 51. Leordeanu, M., Sukthankar, R., Sminchisescu, C.: ‘Efficient closed-form solution to generalized boundary detection’. European Conf. on Computer Vision (ECCV), 2012, pp. 516–529.
    11. 11)
      • 3. Wedel, A., Pock, T., Zach, C., et al: ‘An improved algorithm for TV-L 1 optical flow’. Statistical and Geometrical Approaches to Visual Motion Analysis, 2009.
    12. 12)
      • 27. Xu, L., Jia, J., Matsushita, Y.: ‘Motion detail preserving optical flow estimation’. Computer Vision and Pattern Recognition (CVPR), 2010.
    13. 13)
      • 19. Furukawa, Y., Ponce, J.: ‘Accurate, dense, and robust multi-view stereopsis’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (8), pp. 1362–1376.
    14. 14)
      • 30. Bookstein, F.L.: ‘Principal warps: thin-plate splines and the decomposition of deformations’, IEEE Trans. Pattern Anal. Mach. Intell., 1989, 11, (6), pp. 567–585.
    15. 15)
      • 16. Hirschmuller, H.: ‘Accurate and efficient stereo processing by semi-global matching and mutual information’. Computer Vision and Pattern Recognition (CVPR), 2005, vol. 2, pp. 807–814.
    16. 16)
      • 1. Horn, B.K.P., Schunck, B.G.: ‘Determining optical flow’, Artif. Intell., 1981, 17, pp. 185–203.
    17. 17)
      • 31. Lee, S., Wolberg, G., Chwa, K., et al: ‘Image metamorphosis with scattered feature constraints’, Vis. Comput. Graph., 1996, 2, (4), pp. 337–354.
    18. 18)
      • 9. Lowe, D.G.: ‘Distinctive image features from scale-invariant keypoints’, Int. J. Comput. Vis., 2004, 60, (2), pp. 91–110.
    19. 19)
      • 44. Weinzaepfel, P., Revaud, J., Harchaoui, Z., et al: ‘DeepFlow: LARGE displacement optical flow with deep matching’. Int. Conf. on Computer Vision (ICCV), 2013.
    20. 20)
      • 10. Rosten, E., Porter, R., Drummond, T.: ‘Faster and better: a machine learning approach to corner detection’. PAMI, 2010.
    21. 21)
      • 13. Grompone von Gioi, R., Jakubowicz, J., Morel, J.-M., et al: ‘LSD: a line segment detector’, Image Process. On Line, 2012, 2, pp. 35–55.
    22. 22)
      • 35. Pock, T., Chambolle, A.: ‘Diagonal preconditioning for first order primal-dual algorithms in convex optimization’. Int. Conf. on Computer Vision (ICCV), 2011, pp. 1762–1769.
    23. 23)
      • 24. Tran, Q.-H., Chin, T.-J., Carneiro, G., et al: ‘In defence of RANSAC for outlier rejection in deformable registration’. European Conf. on Computer Vision (ECCV), 2012, pp. 274–287.
    24. 24)
      • 40. Geiger, A., Lenz, P., Urtasun, R.: ‘Are we ready for autonomous driving? The KITTI vision benchmark suite’. CVPR, 2012.
    25. 25)
      • 29. Rother, C., Kolmogorov, V., Lempitsky, V., et al: ‘Optimizing binary MRFs via extended roof duality’. Computer Vision and Pattern Recognition (CVPR), 2007, pp. 1–8.
    26. 26)
      • 36. Bredies, K.: ‘Recovering piecewise smooth multichannel images by minimization of convex functional with total generalized variation penalty’. SFB Report, 6, 2012.
    27. 27)
      • 37. Werlberger, M.: ‘Convex approaches for high performance video processing’. PhD thesis, Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria, June 2012.
    28. 28)
      • 17. Mei, X., Sun, X., Zhou, M., et al: ‘On building an accurate stereo matching system on graphics hardware’. Third ICCV Workshop on GPUs for Computer Vision, 2011.
    29. 29)
      • 22. Pilet, J., Lepetit, V., Fua, P.: ‘Fast non-rigid surface detection, registration and realistic augmentation’, Int. J. Comput. Vis., 2008, 76, (2), pp. 109–122.
    30. 30)
      • 38. Eberhart, R.C., Shi, Y.: ‘Comparing inertia weights and constriction factors in particle swarm optimization’. Congress on Evolutionary Computation, 2000, vol. 1, pp. 84–88.
    31. 31)
      • 26. Leordeanu, M., Zanfir, A., Sminchisescu, C.: ‘Locally affine sparse-to-dense matching for motion and occlusion estimation’. Int. Conf. on Computer Vision (ICCV), 2013.
    32. 32)
      • 25. Wills, J., Agarwal, S., Belongie, S.: ‘A feature-based approach for dense segmentation and estimation of large disparity motion’, Int. J. Comput. Vis., 2006, 68, (2), pp. 125–143.
    33. 33)
      • 45. Ferrari, V., Tuytelaars, T., Van Gool, L.: ‘Simultaneous object recognition and segmentation by image exploration’. European Conf. on Computer Vision (ECCV), 2004, pp. 40–54.
    34. 34)
      • 2. Chambolle, A., Pock, T.: ‘A first-order primal-dual algorithm for convex problems with applications to imaging’, J. Math. Imaging Vis., 2011, 40, (1), pp. 120–145.
    35. 35)
      • 49. Salzmann, M., Fua, P.: ‘Reconstructing sharply folding surfaces: A convex formulation’. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 1054–1061.
    36. 36)
      • 48. Prasad, M., Fitzgibbon, A.: ‘Single view reconstruction of curved surfaces’. Computer Vision and Pattern Recognition (CVPR), 2006, vol. 2, pp. 1345–1354.
    37. 37)
      • 14. Liu, C., Yuen, J., Torralba, A.: ‘SIFT flow: dense correspondence across scenes and its applications’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (5), pp. 978–994.
    38. 38)
      • 50. Dosovitskiy, A., Fischer, P., Ilg, E., et al: ‘Flownet: learning optical flow with convolutional networks’, December 2015.
    39. 39)
      • 20. Garg, R., Roussos, A., Agapito, L.: ‘A variational approach to video registration with subspace constraints’, Int. J. Comput. Vis., 2013, 104, (3), pp. 286–314.
    40. 40)
      • 28. Brox, T., Malik, J.: ‘Large displacement optical flow: descriptor matching in variational motion estimation’, IEEE Trans. Pattern Anal. Mach. Intell., 2011, 33, (3), pp. 500–513.
    41. 41)
      • 34. Ochs, P., Chen, Y., Brox, T., et al: ‘iPiano: inertial proximal algorithm for non-convex optimization’, SIAM J. Imaging Sci. (SIIMS), 2014. Preprint, 7, (2), pp. 1388–1419.
    42. 42)
      • 33. Zabih, R., Woodfill, J.: ‘Non-parametric local transforms for computing visual correspondence’. European Conf. on Computer Vision (ECCV), 1994.
    43. 43)
      • 42. Yu, G., Morel, J.-M.: ‘ASIFT: an algorithm for fully affine invariant comparison’, Image Process. On Line, 2011, 1, pp. 1–28.
    44. 44)
      • 12. Wang, L., Neumann, U., You, S.: ‘Wide-baseline image matching using line signatures’. Int. Conf. on Computer Vision (ICCV), 2009.
    45. 45)
      • 41. Butler, D.J., Wulff, J., Stanley, G.B., et al: ‘A naturalistic open source movie for optical flow evaluation’. European Conf. on Computer Vision (ECCV), 2012, pp. 611–625.
    46. 46)
      • 15. Tola, E., Lepetit, V., Fua, P.: ‘Daisy: an efficient dense descriptor applied to wide-baseline stereo’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (5), pp. 815–830.
    47. 47)
      • 47. Fernández Alcantarilla, P., Bartoli, A.: ‘Deformable 3d reconstruction with an object database’. British Machine Vision Conf. (BMVC), 2012, pp. 1–12.
    48. 48)
      • 46. Salzmann, M., Hartley, R., Fua, P.: ‘Convex optimization for deformable surface 3-d tracking’, Int. Conf. on Computer Vision (ICCV), 2007, pp. 1–8.
    49. 49)
      • 8. Harris, C., Stephens, M.: ‘A combined corner and edge detector’. Alvey Vision Conf., 1988.
    50. 50)
      • 7. Ranftl, R., Gehrig, S., Pock, T., et al: ‘Pushing the limits of stereo using variational stereo estimation’. Intelligent Vehicles Symp. (IV), 2012.
    51. 51)
      • 23. Pizarro, D., Bartoli, A.: ‘Feature-based deformable surface detection with self-occlusion reasoning’, Int. J. Comput. Vis., 2012, 97, (1), pp. 54–70.
    52. 52)
      • 5. Sun, D., Roth, S., Black, M.: ‘A quantitative analysis of current practices in optical flow estimation and the principles behind them’, Int. J. Comput. Vis., 2014, 106, (2), pp. 115–137.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2016.0090
Loading

Related content

content/journals/10.1049/iet-cvi.2016.0090
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address