access icon free Optimisation-based training of evolutionary convolution neural network for visual classification applications

Training of the convolution neural network (CNN) is a problem of global optimisation. This study proposed a hybrid modified particle swarm optimisation (MPSO) and conjugate gradient (CG) algorithm for efficient training of CNN. The training involves MPSO–CG to avoid trapping in local minima. Particularly, improvements in the MPSO by introducing a novel approach for control parameters, improved parameters updating criteria, a novel parameter in the velocity update equation, and fusion of the CG allows handling the issues in training CNN. In this study, the authors validate the proposed MPSO algorithm on three benchmark mathematical test functions and also compared with three different variants of the baseline particle swarm optimisation algorithm. Furthermore, the performance of the proposed MPSO–CG is also compared with other training algorithms focusing on the analysis of computational cost, convergence, and accuracy based on a standard problem specific to classification applications on CIFAR-10 dataset and face and skin detection dataset.

Inspec keywords: convolutional neural nets; optimisation; image classification; particle swarm optimisation

Other keywords: training algorithms; baseline particle swarm optimisation algorithm; training CNN; optimisation-based training; visual classification applications; global optimisation; improved parameters; MPSO–CG; velocity update equation; MPSO algorithm; control parameters; evolutionary convolution neural network

Subjects: Neural computing techniques; Optimisation techniques; Knowledge engineering techniques; Computer vision and image processing techniques; Image recognition; Optimisation techniques

References

    1. 1)
      • 39. Rehman, O.U., Yang, S., Khan, S., et al: ‘A quantum particle swarm optimizer with enhanced strategy for global optimization of electromagnetic devices’, IEEE Trans. Magn., 2019, 55, pp. 14.
    2. 2)
      • 27. Yang, J., Yu, K., Gong, Y., et al: ‘Linear spatial pyramid matching using sparse coding for image classification’. 2009 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2009), Miami, FL, USA., 2009, pp. 17941801.
    3. 3)
      • 19. Yamasaki, T., Honma, T., Aizawa, K.: ‘Efficient optimization of convolutional neural networks using particle swarm optimization’. 2017 IEEE Third Int. Conf. on Multimedia Big Data (BigMM), Laguna Hills, California, USA., 2017, pp. 7073.
    4. 4)
      • 34. Tang, Y.: ‘Deep learning using linear support vector machines’, arXiv preprint arXiv:1306.0239, 2013.
    5. 5)
      • 33. Hinton, G.E., Srivastava, N., Krizhevsky, A., et al: ‘Improving neural networks by preventing co-adaptation of feature detectors’, arXiv preprint arXiv:1207.0580, 2012.
    6. 6)
      • 55. Higashi, N., Iba, H.: ‘Particle swarm optimization with Gaussian mutation’. Proc. 2003 IEEE Swarm Intelligence Symp. (SIS'03), Indiana, USA., 2003, pp. 7279.
    7. 7)
      • 40. Rehman, O.U., Rehman, S.U., Tu, S., et al: ‘A quantum particle swarm optimization method with fitness selection methodology for electromagnetic inverse problems’, IEEE Access, 2018, 6, pp. 63 15563 163.
    8. 8)
      • 31. Jarrett, K., Kavukcuoglu, K., LeCun, Y., et al: ‘What is the best multi-stage architecture for object recognition?'. 2009 IEEE 12th Int. Conf. on Computer Vision, Kyoto, Japan, 2009, pp. 21462153.
    9. 9)
      • 14. Liu, H., Tian, H.-q., Chen, C., et al: ‘An experimental investigation of two wavelet-MLP hybrid frameworks for wind speed prediction using GA and PSO optimization’, Int. J. Electr. Power Energy Syst., 2013, 52, pp. 161173.
    10. 10)
      • 26. Zeiler, M.D., Fergus, R.: ‘Visualizing and understanding convolutional networks’. European Conf. on Computer Vision, Zurich, Switzerland, 2014, pp. 818833.
    11. 11)
      • 2. Rehman, S.U., Tu, S., Huang, Y., et al: ‘CSFL: a novel unsupervised convolution neural network approach for visual pattern classification’, AI Commun., 2017, 30, (5), pp. 311324.
    12. 12)
      • 44. Ngiam, J., Chen, Z., Chia, D., et al: ‘Tiled convolutional neural networks’, in Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., et al (Eds.): Advances in Neural Information Processing Systems 23, (Curran Associates, Inc., 2010), pp. 12791287.
    13. 13)
      • 22. ur Rehman, S., Tu, S., Waqas, M., et al: ‘Unsupervised pre-trained filter learning approach for efficient convolution neural network’, Neurocomputing, 2019, 365, pp. 171190.
    14. 14)
      • 48. Coates, A., Ng, A., Lee, H.: ‘An analysis of single-layer networks in unsupervised feature learning’. Proc. 14th Int. Conf. on Artificial Intelligence and Statistics, FL, USA., 2011, pp. 215223.
    15. 15)
      • 28. Boureau, Y.-L., Ponce, J., LeCun, Y.: ‘A theoretical analysis of feature pooling in visual recognition’. Proc. 27th Int. Conf. on Machine Learning (ICML-10), Haifa, Israel, 2010, pp. 111118.
    16. 16)
      • 4. Ding, C., Tao, D.: ‘Robust face recognition via multimodal deep face representation’, IEEE Trans. Multimed., 2015, 17, (11), pp. 20492058.
    17. 17)
      • 21. LeCun, Y., Bottou, L., Bengio, Y., et al: ‘Gradient-based learning applied to document recognition’, Proc. IEEE, 1998, 86, (11), pp. 22782324.
    18. 18)
      • 54. Zhan, Z.-H., Zhang, J., Li, Y., et al: ‘Adaptive particle swarm optimization’, IEEE Trans. Syst. Man Cybern. B, Cybern., 2009, 39, (6), pp. 13621381.
    19. 19)
      • 5. Rehman, S.u., Tu, S., Huang, Y., et al: ‘Optimization of CNN through novel training strategy for visual classification problems’, Entropy, 2018, 20, (4), p. 290.
    20. 20)
      • 1. Zhang, K., Zhang, Z., Li, Z., et al: ‘Joint face detection and alignment using multitask cascaded convolutional networks’, IEEE Signal Process. Lett., 2016, 23, (10), pp. 14991503.
    21. 21)
      • 29. Huang, F.J., Boureau, Y.-L., LeCun, Y., et al: ‘Unsupervised learning of invariant feature hierarchies with applications to object recognition’. 2007 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR'07), Minneapolis, USA., 2007, pp. 18.
    22. 22)
      • 13. Ludermir, T.B., De Oliveira, W.R.: ‘Particle swarm optimization of MLP for the identification of factors related to common mental disorders’, Expert Syst. Appl., 2013, 40, (11), pp. 46484652.
    23. 23)
      • 47. Yu, K., Zhang, T.: ‘Improved local coordinate coding using local tangents'. ICML'10: Proc. of the 27th Int. Conf. on Machine Learning, Haifa, Israel, 2010, pp. 12151222.
    24. 24)
      • 18. Wang, B., Sun, Y., Xue, B., et al: ‘A hybrid GA-PSO method for evolving architecture and short connections of deep convolutional neural networks’, arXiv preprint arXiv:1903.03893, 2019.
    25. 25)
      • 9. Liu, K.-C., Hsu, C.-C., Wang, W.-Y., et al: ‘Real-time facial expression recognition based on CNN’. 2019 Int. Conf. on System Science and Engineering (ICSSE), Dong Hoi, Vietnam, 2019, pp. 120123.
    26. 26)
      • 3. Low, C.-Y., Teoh, A.B.-J., Toh, K.-A.: ‘Stacking PCANet+: an overly simplified ConvNets baseline for face recognition’, IEEE Signal Process. Lett, 2017, 24, pp. 15811585.
    27. 27)
      • 15. Warsito, B., Yasin, H., Prahutama, A.: ‘Particle swarm optimization versus gradient based methods in optimizing neural network’. J. Phys. Conf. Ser., Central Java, Indonesia, 2019, 1217, pp. 012101.
    28. 28)
      • 37. Kennedy, J.: ‘Particle swarm optimization’ in ‘Encyclopedia of machine learning’ (Springer, 2011), pp. 760766.
    29. 29)
      • 36. Hatipoglu, N., Bilgin, G.: ‘Classification of histopathological images using convolutional neural network’. 2014 4th Int. Conf. on Image Processing Theory, Tools and Applications (IPTA), Paris, France, 2014, pp. 16.
    30. 30)
      • 7. Rehman, S.U., Tu, S., Huang, Y., et al: ‘A benchmark dataset and learning high-level semantic embeddings of multimedia for cross-media retrieval’, IEEE Access, 2018, 6, pp. 67 17667 188.
    31. 31)
      • 12. LeCun, Y., Boser, B.E., Denker, J.S., et al: ‘Handwritten digit recognition with a back-propagation network’, in Touretzky, D.S. (Ed.): Advances in Neural Information Processing Systems 2, (Morgan-Kaufmann, 1990), pp. 396404.
    32. 32)
      • 8. Rehman, S.U., Tu, S., Huang, Y., et al: ‘Face recognition: a novel un-supervised convolutional neural network method’. IEEE Int. Conf. of Online Analysis and Computing Science (ICOACS), Chongqing, China, 2016, pp. 139144.
    33. 33)
      • 51. Seha, S.N.A., Hatzinakos, D.: ‘Human recognition using transient auditory evoked potentials: a preliminary study’, IET Biometrics, 2018, 7, (3), pp. 242250.
    34. 34)
      • 24. Nair, V., Hinton, G.E.: ‘Rectified linear units improve restricted Boltzmann machines’. Proc. 27th int. Conf. on machine learning (ICML-10), Haifa, Israel, 2010, pp. 807814.
    35. 35)
      • 25. Phung, S.L., Bouzerdoum, A.: ‘A pyramidal neural network for visual pattern recognition’, IEEE Trans. Neural Netw., 2007, 18, (2), pp. 329343.
    36. 36)
      • 20. da Silva, G.L.F., Valente, T.L.A., Silva, A.C., et al: ‘Convolutional neural network-based PSO for lung nodule false positive reduction on CT images’, Comput. Methods Programs Biomed., 2018, 162, pp. 109118.
    37. 37)
      • 42. Krizhevsky, A., Hinton, G.: ‘Learning multiple layers of features from tiny images’, Citeseer, Tech. Rep., 2009.
    38. 38)
      • 38. Rehman, O.U., Tu, S., Rehman, S.U., et al: ‘Design optimization of electromagnetic devices using an improved quantum inspired particle swarm optimizer', Appl. Comput. Electromagn. Soc. J., 2018, 33, (9).
    39. 39)
      • 45. Bo, L., Ren, X., Fox, D.: ‘Kernel descriptors for visual recognition’, in J.D., Lafferty, C.K.I., Williams, Shawe-Taylor, J., Shawe-Taylor, J., et al (Eds.): Advances in Neural Information Processing Systems 23, (Curran Associates, Inc., 2010, pp. 244252.
    40. 40)
      • 41. Fletcher, R., Reeves, C.M.: ‘Function minimization by conjugate gradients’, Comput. J., 1964, 7, (2), pp. 149154.
    41. 41)
      • 32. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’, arXiv preprint arXiv:1409.1556, 2014.
    42. 42)
      • 50. Khamsemanan, N., Nattee, C., Jianwattanapaisarn, N.: ‘Human identification from freestyle walks using posture-based gait feature’, IEEE Trans. Inf. Forensics Sec., 2017, 13, (1), pp. 119128.
    43. 43)
      • 46. Chan, T.-H., Jia, K., Gao, S., et al: ‘PCANet: a simple deep learning baseline for image classification?’, IEEE Trans. Image Process., 2015, 24, (12), pp. 50175032.
    44. 44)
      • 23. Fukushima, K., Miyake, S.: ‘Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position’, Pattern Recognit., 1982, 15, (6), pp. 455469.
    45. 45)
      • 6. Bulan, O., Kozitsky, V., Ramesh, P., et al: ‘Segmentation- and annotation-free license plate recognition with deep localization and failure identification’, IEEE Trans. Intell. Transp. Syst., 2017, 18, (9), pp. 23512363.
    46. 46)
      • 17. Chhabra, Y., Varshney, S., Wadhwa, A.: ‘Hybrid particle swarm training for convolution neural network (CNN)’. 2017 Tenth Int. Conf. on Contemporary Computing (IC3), Noida, India, August 2017, pp. 13.
    47. 47)
      • 16. Zhu, F., Xu, C.: ‘Particle swarm hybridize with Gaussian process regression for displacement prediction’. 2010 IEEE Fifth Int. Conf. on Bio-Inspired Computing: Theories and Applications (BIC-TA), Liverpool, UK., 2010, pp. 522525.
    48. 48)
      • 49. Krizhevsky, A.: ‘Cuda-convnet’, code.google.com/p/cudaconvnet, 2014.
    49. 49)
      • 10. Lu, Q., Liu, Y., Huang, J., et al: ‘License plate detection and recognition using hierarchical feature layers from CNN’, Multimedia Tools Appl., 2019, 78, (11), pp. 15 66515 680.
    50. 50)
      • 35. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ‘ImageNet classification with deep convolutional neural networks’. Advances in Neural Information Processing Systems, NY, USA., 2012, pp. 10971105.
    51. 51)
      • 52. Damer, N., Opel, A., Nouak, A.: ‘CMC curve properties and biometric source weighting in multi-biometric score-level fusion’. 17th Int. Conf. on Information Fusion (FUSION), Salamanca, Spain, 2014, pp. 16.
    52. 52)
      • 30. Wang, T., Wu, D.J., Coates, A., et al: ‘End-to-end text recognition with convolutional neural networks’. 2012 21st Int. Conf. on Pattern Recognition (ICPR), Tsukuba, Japan, 2012, pp. 33043308.
    53. 53)
      • 43. Phung, S.L., Bouzerdoum, A., Chai, D.: ‘Skin segmentation using color pixel classification: analysis and comparison’, IEEE Trans. Pattern Anal. Mach. Intell., 2005, 27, (1), pp. 148154.
    54. 54)
      • 11. Wang, L., Liao, J., Xu, C.: ‘Vehicle detection based on drone images with the improved faster R-CNN’. Proc. 2019 11th Int. Conf. on Machine Learning and Computing, Zhuhai, China, 2019, pp. 466471.
    55. 55)
      • 53. DeCann, B., Ross, A.: ‘Relating ROC and CMC curves via the biometric menagerie’. 2013 IEEE Sixth Int. Conf. on Biometrics: Theory, Applications and Systems (BTAS), Washington, DC, USA., 2013, pp. 18.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2019.0506
Loading

Related content

content/journals/10.1049/iet-cvi.2019.0506
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading