End-to-end training algorithm for conceptor-based neural networks

Guangwu Qian; Lei Zhang; Qianjun Zhang

End-to-end training algorithm for conceptor-based neural networks

View Fulltext

Author(s): Guangwu Qian¹ ; Lei Zhang¹ ; Qianjun Zhang¹
- Affiliations: 1: Machine Intelligence Laboratory, College of Computer Science, Sichuan University , Chengdu , People's Republic of China
Source: Volume 54, Issue 15, 26 July 2018, p. 924 – 926
DOI: 10.1049/el.2018.0033 , Print ISSN 0013-5194, Online ISSN 1350-911X

Received 12/03/2018, Published 19/06/2018

Since a conceptor, achieving direction-selective damping of high-dimensional network signals, usually takes the form of a projection matrix and is deduced analytically, a conceptor-based neural network is thought to be untrainable with backpropagation and gradient-descent algorithms from end to end. It limits the application of conceptors. To address this issue, an algorithm is proposed to train conceptor-based neural networks from end to end with gradient-descent algorithms. To the best of the authors’ knowledge, it is the first work of such an end-to-end training algorithm. To develop this algorithm, a softmax-like loss function involved with conceptors is constructed empirically. Based on this loss function, corresponding gradients are deduced by using backpropagation method so that it is possible to train conceptor neural networks from end to end with a gradient-descent algorithm. Several experiments are conducted to show the feasibility and effectiveness of the proposed training algorithm.

References

1. 1)
  - 8. Qian, G., Zhang, L.: ‘A simple feedforward convolutional conceptor neural network for classification’, Appl. Soft Comput., 2017, doi: 10.1016/j.asoc.2017.08.016.
2. 2)
  - 7. Jaeger, H.: ‘Using conceptors to manage neural long-term memories for temporal patterns’, J. Mach. Learn. Res., 2017, 18, (13), pp. 1–43.
3. 3)
  - 3. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ‘Imagenet classification with deep convolutional neural networks’. Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, December 2012, pp. 1097–1105.
4. 4)
  - 38. Bengio, Y., Simard, P., Frasconi, P.: ‘Learning long-term dependencies with gradient descent is difficult’, IEEE Trans. Neutral Netw.1994, 5, (2), pp. 157–166 (doi: 10.1109/72.279181).
5. 5)
  - 12. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’, arXiv preprint arXiv:1409.1556, 2014.
6. 6)
  - 20. Lecun, Y., Bottou, L., Bengio, Y., et al: ‘Gradient-based learning applied to document recognition’, Proc. IEEE, 1998, 86, (11), pp. 2278–2324 (doi: 10.1109/5.726791).
7. 7)
  - 1. Hinton, G., Osindero, S.: ‘A fast learning algorithm for deep belief nets’, Neural Comput., 2006, 18, (7), pp. 1527–1554 (doi: 10.1162/neco.2006.18.7.1527).
8. 8)
  - 4. Wan, L., Zeiler, M., Zhang, S., et al: ‘Regularization of neural networks using dropconnect’. Proc. of the 30th Int. Conf. on Machine Learning (ICML-13), Atlanta, GA, USA, June 2013, pp. 1058–1066.
9. 9)
  - 5. Hinton, G.E., Salakhutdinov, R.R.: ‘Reducing the dimensionality of data with neural networks’, Science, 2006, 313, (5786), pp. 504–507 (doi: 10.1126/science.1127647).
10. 10)
  - 23. Fei-Fei, L., Fergus, R., Perona, P.: ‘Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories’, Comput. Vis. Image Underst., 2007, 106, (1), pp. 59–70 (doi: 10.1016/j.cviu.2005.09.012).
11. 11)
  - 5. Rumelhart, D.E., Hinton, G.E., Williams, R.J., et al: ‘Learning representations by back-propagating errors’, Cogn. Model., 1988, 5, (3), p. 1.
12. 12)
  - 15. Russakovsky, O., Deng, J., Su, H., et al: ‘Imagenet large scale visual recognition challenge’, Int. J. Comput. Vis., 2015, 115, (3), pp. 211–252 (doi: 10.1007/s11263-015-0816-y).
13. 13)
  - 9. Qian, G., Zhang, L., Zhang, Q.: ‘Fast conceptor classifier in pre-trained neural networks for visual recognition’. Int. Symp. on Neural Networks, Sapporo, Hakodate, and Muroran, Hokkaido, Japan, June 2017, pp. 290–298.

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

End-to-end training algorithm for conceptor-based neural networks

References

Related content