Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free Convolutional neural networks analysed via inverse problem theory and sparse representations

© The Institution of Engineering and Technology
Received 08/06/2018, Accepted 02/10/2018, Revised 10/09/2018, Published 04/10/2018

Inverse problems in imaging such as denoising, deblurring, superresolution have been addressed for many decades. In recent years, convolutional neural networks (CNNs) have been widely used for many inverse problem areas. Although their indisputable success, CNNs are not mathematically validated as to how and what they learn. In this study, the authors prove that during training, CNN elements solve for inverse problems which are optimum solutions stored as CNN neuron filters. They discuss the necessity of mutual coherence between CNN layer elements in order for a network to converge to the optimum solution. They prove that required mutual coherence can be provided by the usage of residual learning and skip connections. They have set rules over training sets and depth of networks for better convergence, i.e. performance. They have experimentally validated theoretical assertions.

Inspec keywords: inverse problems; learning (artificial intelligence); filtering theory; feedforward neural nets; image resolution; cellular neural nets; image denoising; image restoration

Other keywords: training sets; residual learning; CNN neuron filters; convolutional neural networks; skip connections; mutual coherence; sparse representations; CNN layer elements; CNN elements; inverse problem theory

Subjects: Computer vision and image processing techniques; Mathematical analysis; Neural computing techniques; Knowledge engineering techniques

References

    1. 1)
      • 9. Mao, X., Shen, C., Yang, Y.: ‘Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections’. Advances in Neural Information Processing Systems 29: Annual Conf. Neural Information Processing Systems, Barcelona, 2016, pp. 2802–2810.
    2. 2)
      • 1. LeCun, Y., Bengio, Y., Hinton, G.E.: ‘Deep learning’, Nature, 2015, 521, (7553), pp. 436–444.
    3. 3)
      • 5. Yang, Y., Sun, J., Li, H., et al: ‘Deep admm-net for compressive sensing MRI’. Advances in Neural Information Processing Systems 29: Annual Conf. on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016, pp. 10–18.
    4. 4)
      • 28. Mallat, S.: ‘Understanding deep convolutional networks’, CoRR, 2016, available at: http://arxiv.org/abs/1601.04920.
    5. 5)
      • 29. Papyan, V., Sulam, J., Elad, M.: ‘Working locally thinking globally: theoretical guarantees for convolutional sparse coding’, IEEE Trans. Signal Process., 2017, 65, (21), pp. 5687–5701.
    6. 6)
      • 11. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution as sparse representation of raw image patches’. 2008 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR 2008), Anchorage, June 2008.
    7. 7)
      • 23. Maas, A.L., Hannun, A.Y., Ng, A.Y.: ‘Rectifier nonlinearities improve neural network acoustic models’. ICML Workshop on Deep Learning for Audio, Speech and Language Processing, Atlanta, 2013.
    8. 8)
      • 18. Combettes, P.L., Wajs, V.R.: ‘Signal recovery by proximal forward-backward splitting’, Multiscale. Model. Simul., 2005, 4, (4), pp. 1168–1200.
    9. 9)
      • 25. Bruna, J., Sprechmann, P., LeCun, Y.: ‘Super-resolution with deep convolutional sufficient statistics’, CoRR, 2015, available at http://arxiv.org/abs/1511.05666.
    10. 10)
      • 8. Papyan, V., Romano, Y., Elad, M.: ‘Convolutional neural networks analyzed via convolutional sparse coding’, CoRR, 2016, 18, pp. 83:1–83:52.
    11. 11)
      • 12. Yang, J., Wright, J., Huang, T.S., et al: ‘Image super-resolution via sparse representation’, Trans. Image Process., 2010, 19, (11), pp. 2861–2873.
    12. 12)
      • 21. Zhang, K., Zuo, W., Chen, Y., et al: ‘Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising’, IEEE Trans. Image Process., 2017, 26, (7), pp. 3142–3155.
    13. 13)
      • 27. Daubechies, I., Defrise, M., De Mol, C.: ‘An iterative thresholding algorithm for linear inverse problems with a sparsity constraint’, Commun. Pure Appl. Math., 2004, 57, (11), pp. 1413–1457.
    14. 14)
      • 2. McCann, M.T., Jin, K.H., Unser, M.: ‘Convolutional neural networks for inverse problems in imaging: a review’, IEEE Signal Process. Mag., 2017, 34, (6), pp. 85–95.
    15. 15)
      • 19. Bronstein, A.M., Sprechmann, P., Sapiro, G.: ‘Learning efficient structured sparse models’. Proc. 29th Int. Conf. Machine Learning (ICML), Edinburgh, 2012.
    16. 16)
      • 26. Shi, W., Caballero, J., Huszar, F., et al: ‘Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network’. Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016, pp. 1874–1883.
    17. 17)
      • 14. Timofte, R., Smet, V.D., Gool, L.J.V.: ‘A+: adjusted anchored neighborhood regression for fast super-resolution’. ACCV (4), Singapore, 2014 (LNCS, 9006), pp. 111–126.
    18. 18)
      • 20. Schuler, C.J., Hirsch, M., Harmeling, S., et al: ‘Learning to deblur’, IEEE Trans. Pattern Anal. Mach. Intell., 2016, 38, (7), pp. 1439–1451.
    19. 19)
      • 24. Glorot, X., Bordes, A., Bengio, Y. ‘Deep sparse rectifier neural networks’. Proc. Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, April 2011, vol. 15, JMLR.org, pp. 315–323.
    20. 20)
      • 6. Dong, C., Loy, C.C., He, K., et al: ‘Learning a deep convolutional network for image super-resolution’. European Conf. on Computer Vision (ECCV) (4), Zurich, September 2014 (LNCS, 8692), pp. 184–199.
    21. 21)
      • 22. Gabay, D., Mercier, B.: ‘A dual algorithm for the solution of nonlinear variational problems via finite element approximation’, Comput. Math. Appl., 1976, 2, (1), pp. 17–40.
    22. 22)
      • 7. Bengio, Y., Courville, A.C., Vincent, P.: ‘Representation learning: A review and new perspectives’, IEEE Trans. Pattern Anal. Mach. Intell., 2013, 35, (8), pp. 1798–1828.
    23. 23)
      • 13. Timofte, R., Smet, V.D., Gool, L.J.V.: ‘Anchored neighborhood regression for fast example-based super-resolution’. Int. Conf. Computer Vision (ICCV), Portland, 2013, pp. 1920–1927.
    24. 24)
      • 17. Aharon, M., Elad, M., Bruckstein, A.: ‘Ksvd: an algorithm for designing overcomplete dictionaries for sparse representation’, Trans. Signal Process., 2006, 54, (11), pp. 4311–4322.
    25. 25)
      • 4. Gregor, K., LeCun, Y.: ‘Learning fast approximations of sparse coding’. Proc. 27th Int. Conf. Machine Learning (ICML-10), Haifa, 2010, pp. 399–406.
    26. 26)
      • 10. Kim, J., Lee, J.K., Lee, K.M.: ‘Accurate image super-resolution using very deep convolutional networks’. Computer Vision and Pattern Recognition (CVPR), Las Vegas, 2016, pp. 1646–1654.
    27. 27)
      • 15. Romano, Y., Isidoro, J., Milanfar, P.: ‘RAISR: rapid and accurate image super resolution’, IEEE Trans. Comput. Imag., 2017, 3, (1), pp. 110–125.
    28. 28)
      • 3. Lucas, A., Iliadis, M., Molina, R., et al: ‘Using deep neural networks for inverse problems in imaging: beyond analytical methods’, IEEE Signal Process. Mag., 2018, 35, (1), pp. 20–36.
    29. 29)
      • 16. Chen, Y., Pock, T.: ‘Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration’, IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, (6), pp. 1256–1272.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-spr.2018.5220
Loading

Related content

content/journals/10.1049/iet-spr.2018.5220
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address