access icon free Multi-frame super-resolution algorithm using common vector approach

© The Institution of Engineering and Technology
Received 15/03/2018, Accepted 04/09/2018, Revised 30/07/2018, Published 07/09/2018

Super-resolution (SR) applications aim to use information within one or more low-resolution (LR) image(s) to obtain high-resolution (HR) image(s). LR images might well be the consecutive frames of a video sequence. When multiple LR images are used as input, motion estimation (ME) between portions of images is an important step of the solution for this ill-posed problem. In this study, the authors employed translational optical flow for ME, followed by common vector approach (CVA) for HR image reconstruction from multiple sources. CVA provides a way to reduce outliers caused by noise, occlusion, shadows and incorrect ME. HR image blocks are obtained by combining common and difference vectors of the blocks’ class that are handled separately. Noise in difference vectors is reduced by a known noise reduction method before combining. Separate handling of common and difference parts guarantees better results and greatly reduce artefacts. Compared to the state-of-the-art, experimental results confirm the authors’ achievement by visual, peak signal-to-noise ratio and structural similarity index measures criteria.

Inspec keywords: image sequences; vectors; image resolution; motion estimation; image denoising; image reconstruction

Other keywords: high-resolution image; multiframe super-resolution algorithm; multiple sources; multiple LR images; HR image reconstruction; HR image blocks; motion estimation; translational optical flow; common vector approach; low-resolution image; difference vectors; peak signal-to-noise ratio; visual signal-to-noise ratio; video sequence; consecutive frames; super-resolution applications; noise reduction method; shadows

Subjects: Algebra; Optical, image and video signal processing; Computer vision and image processing techniques; Algebra

References

    1. 1)
      • 28. Guo, K., Yang, X., Lin, W., et al: ‘Learning-based super-resolution method with a combining of both global and local constraints’, IET Image Process., 2012, 6, (4), pp. 337–344.
    2. 2)
      • 6. Borman, S., Stevenson, R.L.: ‘Super-resolution from image sequences – a review’. Midwest Symp. on Circuits and Sys., Notre Dame, USA, 1998, pp. 374–378.
    3. 3)
      • 31. Kaveh, A., Ezzatollah, S.: ‘Single-image super resolution using evolutionary sparse coding technique’, IET Image Process., 2017, 11, (1), pp. 13–21.
    4. 4)
      • 40. Wang, Y., Zhang, Y., Ostermann, J.: ‘Video processing and com’ (Prentice-Hall, Upper Saddle River, NJ, 2001).
    5. 5)
      • 4. Li, X., Orchrard, M.: ‘New edge-directed interpolation’, IEEE Trans. Image Process., 2001, 10, (10), pp. 1521–1527.
    6. 6)
      • 41. Gulmezoglu, M.B., Barkana, A.: ‘Text-dependent speaker recognition by using Gram-Schmidt orthogonalization method’. Proc. IASTED Int. Conf. Signal Processing and Applications, Canary Islands, Spain, 1998, pp. 438–440.
    7. 7)
      • 23. Zhang, L., Zhang, H., Shen, H., et al: ‘A super-resolution reconstruction algorithm for surveillance images’, Signal Process., 2010, 90, (3), pp. 848–859.
    8. 8)
      • 30. Kang, L.W, Hsu, C.C., Zhuang, B., et al: ‘Learning-based joint super-resolution and deblocking for a highly compressed image’, IEEE Trans. Image Process., 2015, 17, (7), pp. 921–934.
    9. 9)
      • 9. Zitova, B., Flusser, J.: ‘Image registration methods: a survey’, Image Vision Comput., 2003, 22, pp. 977–1000.
    10. 10)
      • 7. Park, S.C., Park, M.K., Kang, M.C.: ‘Super-resolution image reconstruction: a technical overview’, IEEE Signal Process. Mag., 2003, 20, (3), pp. 21–36.
    11. 11)
      • 3. Turkowski, K.: ‘Filters for common re-sampling tasks’, in Glassner, A.S. (Ed.): ‘Graphics gems’ (Academic Press Professional, Cambridge, USA, 1990), pp. 147–165.
    12. 12)
      • 37. Kappeler, A., Yoo, S., Dai, Q., et al: ‘Video super-resolution with convolutional neural networks’, IEEE Trans. Comput. Imaging, 2016, 2, (2), pp. 109–122.
    13. 13)
      • 39. Lucas, B., Kanade, T.: ‘An iterative image registration technique with an application to stereo vision’. Proc. DARPA Image Understanding Workshop, Vancouver, Canada, 1981, pp. 121–130.
    14. 14)
      • 26. Su, H., Tang, L., Wu, Y., et al: ‘Spatially adaptive block-based super-resolution’, IEEE Trans. Image Process., 2012, 21, (3), pp. 1031–1045.
    15. 15)
      • 35. Dong, C., Loy, C.C., He, K., et al: ‘Learning a deep convolutional network for image super-resolution’. Proc. IEEE European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 184–199.
    16. 16)
      • 24. Protter, M., Elad, M., Takeda, H., et al: ‘Generalizing the nonlocal-means to superresolution reconstruction’, IEEE Trans. Image Process., 2009, 18, (1), pp. 36–51.
    17. 17)
      • 21. Ji, H., Fermuller, C.: ‘Robust wavelet-based super-resolution reconstruction: theory and algorithm’, IEEE Trans. Pattern Anal. Mach. Intell., 2009, 31, (4), pp. 649–660.
    18. 18)
      • 16. Liu, C., Sun, D.: ‘On Bayesian adaptive video super resolution’, IEEE Trans. Pattern Anal. Mach. Intell., 2014, 36, (2), pp. 346–360.
    19. 19)
      • 32. Mousavi, H.S., Monga, V.: ‘Sparsity-based color image super resolution via exploiting cross channel constraints’, IEEE Trans. Image Process., 2017, 26, (11), pp. 5094–5106.
    20. 20)
      • 13. Lucchese, L., Cortelazzo, G.M.: ‘A noise-robust frequency domain technique for estimating planar roto-translations’, IEEE Trans. Signal Process., 2000, 48, (6), pp. 1769–1786.
    21. 21)
      • 53. Rudin, L., Osher, S., Fatemi, E.: ‘Nonlinear total variation based noise removal algorithms’, Phys. D: Nonlinear Phenom., 1992, 60, pp. 259–268.
    22. 22)
      • 47. Oja, E.: ‘Subspace methods of pattern Recognition’ (Research Studies Press, New York, Wiley, 1983).
    23. 23)
      • 33. Kumar, N., Amit, S.: ‘Fast learning-based single image super-resolution’, IEEE Trans. Multimedia, 2016, 18, (8), pp. 1504–1515.
    24. 24)
      • 14. Schultz, R.R., Meng, L., Stevenson, R.L.: ‘Subpixel motion estimation for superresolution image sequence enhancement’, J. Vis. Commun. Image Represent., 1998, 9, (1), pp. 38–50.
    25. 25)
      • 42. Gulmezoglu, M.B., Dzhafarov, V., Keskin, M., et al: ‘A novel approach to isolated word recognition’, IEEE Trans. Speech Audio Process., 1999, 7, (6), pp. 620–628.
    26. 26)
      • 46. Gulmezoglu, M.B., Ergin, S.: ‘An approach for bearing fault detection in electrical motors’, Eur. Trans. Electr. Power, 2007, 17, (6), pp. 628–641.
    27. 27)
      • 19. Zomet, A., Rav-Acha, A., Peleg, S.: ‘Robust super-resolution’. IEEE Comp. Society Conf. Computer Vision and Pattern Recognition (CVPR 2001), Kauai, USA, 2001, pp. 645–650.
    28. 28)
      • 48. Landgrebe, D.A.: ‘Hyperspectral image data analysis’, IEEE Signal Process. Mag., 2002, 19, (1), pp. 17–28.
    29. 29)
      • 44. Çevikalp, H., Neamtu, M., Wilkes, M., et al: ‘Discriminative common vectors for face recognition’, IEEE Trans. Pattern Anal. Mach. Intell., 2005, 27, (1), pp. 4–13.
    30. 30)
      • 45. Özkan, K., Seke, E.: ‘Image denoising using common vector approach’, IET Image Process., 2015, 9, (8), pp. 709–715.
    31. 31)
      • 36. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016, pp. 770–778.
    32. 32)
      • 18. Keren, D., Peleg, S., Brada, R.: ‘Image sequence enhancement using sub-pixel displacement’. IEEE Conf. Computer Vision and Pattern Recognition, Ann Arbor, USA, 1988, pp. 742–746.
    33. 33)
      • 20. Liu, H.C., Feng, Y., Sun, G.Y.: ‘Wavelet domain image super resolution reconstruction based on image pyramids and cycle spinning’, J. Phys., Conf. Ser., 2006, 48, pp. 4117–4121.
    34. 34)
      • 50. Donoho, D.L., Johnstone, I.M.: ‘Ideal spatial adaptation via wavelet shrinkage’, Biometrika, 1994, 81, pp. 425–455.
    35. 35)
      • 49. Gulmezoglu, M.B., Dzhafarov, V., Edizkan, R., et al: ‘The common vector approach and its comparison with other subspace methods in case of sufficient data’, Comput. Speech Lang., 2007, 21, (2), pp. 266–281.
    36. 36)
      • 22. Farsiu, S., Robinson, M., Elad, M., et al: ‘Fast and robust multi-frame superresolution’, IEEE Trans. Image Process., 2004, 13, (10), pp. 1327–1344.
    37. 37)
      • 8. Brown, L.G.: ‘A survey of image registration techniques’, ACM Comput. Surv., 1992, 24, pp. 325–376.
    38. 38)
      • 17. Irani, M., Rousso, B., Peleg, S.: ‘Computing occluding and transparent motions’, Int. J. Comput. Vision, 1994, 12, (1), pp. 5–16.
    39. 39)
      • 34. Albu, F.: ‘Low complexity image registration techniques based on integral projections’. 2016 Int. Conf. on Systems, Signals and Image Processing (IWSSIP), Bratislava, Slovakia, 2016, pp. 1–4.
    40. 40)
      • 55. Al-Najjar, Y.A.Y., Soong, D.D.C.: ‘Comparison of image quality assessment: Psnr, hvs, ssim, uiqi.’, Int. J. Sci. Eng. Res., 2012, 3, (8), pp. 1–5.
    41. 41)
      • 12. Marcel, B., Briot, M., Murrieta, R.: ‘Calcul de translation et rotation par la transformation de Fourier’, Trait. Signal, 1997, 14, (2), pp. 135–149.
    42. 42)
      • 43. Gulmezoglu, M.B., Dzhafarov, V., Barkana, A.: ‘The common vector approach and its relation to principal component analysis’, IEEE Trans. Speech Audio Process., 2001, 9, (6), pp. 655–662.
    43. 43)
      • 11. Vandewalle, P., Süsstrunk, S.E., Vetterli, M.: ‘A frequency domain approach to registration of aliased images with application to super-resolution’, EURASIP J. Adv. Signal Process., 2006, 2006, pp. 1–14.
    44. 44)
      • 1. Borman, S., Stevenson, R.L.: ‘Simultaneous multi-frame MAP superresolution video enhancement using spatio-temporal priors’. Proc. IEEE Int. Conf. Image Processing, Kobe, Japan, 1999, pp. 469–473.
    45. 45)
      • 38. Stiller, C., Konrad, J.: ‘Estimating motion in image sequences – a tutorial on modeling and computation of 2d motion’, IEEE Signal Process. Mag., 1999, 16, (4), pp. 70–91.
    46. 46)
      • 29. Gao, X., Kaibing, Z., Dacheng, T., et al: ‘Image super-resolution with sparse neighbor embedding’, IEEE Trans. Image Process., 2012, 21, (7), pp. 3194–3205.
    47. 47)
      • 5. Tsai, R.Y., Huang, T.S.: ‘Multi-frame image restoration and registration’, Adv. Comput. Vision Image Process., 1984, 1, (1), pp. 317–339.
    48. 48)
      • 2. Wolberg, G.: ‘Digital image warping’ (IEEE Comp. Society Press, Los Alamitos, California, 1990, 1st edn.) pp. 1–3.
    49. 49)
      • 10. Kim, S.P., Su, W.Y.: ‘Sub-pixel accuracy image registration by spectrum cancellation’. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Minneapolis, USA, 1993, vol. 5, pp. 153–156.
    50. 50)
      • 27. Izadpanahi, S., Demirel, H.: ‘Motion block based video super resolution’, Digital Signal Process., 2013, 23, (5), pp. 1451–1462.
    51. 51)
      • 51. Ajit, R., Anand, R., Arunava, B.: ‘Image denoising using the higher order singular value decomposition’, IEEE Trans. Pattern Anal. Mach. Intell., 2013, 35, (4), pp. 849–862.
    52. 52)
      • 52. Wen, C.K., Chen, J.C., Ting, P.: ‘A shrinkage linear minimum mean square error estimator’, IEEE Signal Process. Lett., 2013, 20, (12), pp. 1179–1182.
    53. 53)
      • 15. Gunturk, B.K., Gevrekci, M.: ‘High-resolution image reconstruction from multiple differently exposed images’, IEEE Signal Process. Lett., 2006, 13, (4), pp. 197–200.
    54. 54)
      • 25. Takeda, H., Milanfar, P., Protter, M.: ‘Super-resolution without explicit sub pixel motion estimation’, IEEE Trans. Image Process., 2009, 18, (9), pp. 1958–1975.
    55. 55)
      • 54. Wang, Z., Bovik, A.C., Sheikh, H.R., et al: ‘Image quality assessment: from error visibility to structural similarity’, IEEE Trans. Image Process., 2004, 13, (4), pp. 2861–2873.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2018.5168
Loading

Related content

content/journals/10.1049/iet-ipr.2018.5168
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading