Mixture separability loss in a deep convolutional network for image classification

Mixture separability loss in a deep convolutional network for image classification

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Image Processing — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

In machine learning, the cost function is crucial because it measures how good or bad a system is. In image classification, well-known networks only consider modifying the network structures and applying cross-entropy loss at the end of the network. However, using only cross-entropy loss causes a network to stop updating weights when all training images are correctly classified. This is the problem of early saturation. This study proposes a novel cost function, called mixture separability loss (MSL), which updates the weights of the network even when most of the training images are accurately predicted. MSL consists of between-class and within-class loss. Between-class loss maximises the differences between inter-class images, whereas within-class loss minimises the similarities between intra-class images. They designed the proposed loss function to attach to different convolutional layers in the network in order to utilise intermediate feature maps. Experiments show that a network with MSL deepens the learning process and obtains promising results with some public datasets, such as Street View House Number, Canadian Institute for Advanced Research, and the authors’ self-collected Inha Computer Vision Lab gender dataset.


    1. 1)
      • 1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ‘Imagenet classification with deep convolutional neural networks’. Conf. Neural Information Processing Systems, Lake Tahoe, Nevada, USA, December 2012, pp. 10971105.
    2. 2)
      • 2. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’. Int. Conf. Learning Representation, San Diego, USA, May 2015.
    3. 3)
      • 3. Chollet, F.: ‘Xception: deep learning with depthwise separable convolutions’. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 2017, pp. 18001807.
    4. 4)
      • 4. Szegedy, C., Vanhoucke, V., Ioffe, S., et al: ‘Rethinking the inception architecture for computer vision’. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June 2016, pp. 28182826.
    5. 5)
      • 5. Szegedy, C., Ioffe, S., Vanhoucke, V., et al: ‘Inception-v4, inception-resnet and the impact of residual connections on learning’.  Association for the Advancement of Artificial Intelligence (AAAI), San Francisco, California, USA, February 2017, pp. 42784284.
    6. 6)
      • 6. Szegedy, C., Liu, W., Jia, Y., et al: ‘Going deeper with convolutions’. IEEE Conf. Computer Vision and Pattern Recognition, Boston, MA, USA, October 2015, pp. 19.
    7. 7)
      • 7. Ioffe, S., Szegedy, C.: ‘Batch normalization: accelerating deep network training by reducing internal covariate shift’, arXiv preprint arXiv:1502.03167, 2015.
    8. 8)
      • 8. He, K., Zhang, X., Ren, S., et al: ‘Identity mappings in deep residual networks’. European Conf. Computer Vision, Amsterdam, Netherlands, October 2016, pp. 630645.
    9. 9)
      • 9. Zagoruyko, S., Komodakis, N.: ‘Wide residual networks’, arXiv preprint arXiv:1605.07146, 2016.
    10. 10)
      • 10. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, June 2016, pp. 770778.
    11. 11)
      • 11. Zhang, K., Sun, M., Han, X., et al: ‘Residual networks of residual networks: multilevel residual networks’, IEEE Trans. Circuits Syst. Video Technol., 2017, 14, (8), pp. 112.
    12. 12)
      • 12. Xie, S., Girshick, R., Dollár, P., et al: ‘Aggregated residual transformations for deep neural networks’. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 2017, pp. 59875995.
    13. 13)
      • 13. Glorot, X., Bengio, Y.: ‘Understanding the difficulty of training deep feedforward neural networks’. Proc. Thirteenth Int. Conf. Artificial Intelligence and Statistics, Sardinia, Italy, 2010, pp. 249256.
    14. 14)
      • 14. Nair, V., Hinton, G.E.: ‘Rectified linear units improve restricted Boltzmann machines’. ICML'10 Proc. 27th Int. Conf. Machine Learning (ICML), Haifa, Israel, June 2010, pp. 807814.
    15. 15)
      • 15. Goodfellow, I.J., Warde-Farley, D., Mirza, M., et al: ‘Maxout networks’, arXiv preprint arXiv:1302.4389, 2013.
    16. 16)
      • 16. Yu, D., Wang, H., Chen, P., et al: ‘Mixed pooling for convolutional neural networks’. Int. Conf. Rough Sets and Knowledge Technology, Shanghai, China, October 2014, pp. 364375.
    17. 17)
      • 17. Hinton, G.E., Srivastava, N., Krizhevsky, A., et al: ‘Improving neural networks by preventing co-adaptation of feature detectors’, arXiv preprint arXiv:1207.0580, 2012.
    18. 18)
      • 18. Wan, L., Zeiler, M., Zhang, S., et al: ‘Regularization of neural networks using dropconnect’. Proc. 30th Int. Conf. Machine Learning, Atlanta, USA, June 2013, pp. 10581066.
    19. 19)
      • 19. Tang, Y.: ‘Deep learning using linear support vector machines’, arXiv preprint arXiv:1306.0239, 2013.
    20. 20)
      • 20. Janocha, K., Czarnecki, W.M.: ‘On loss functions for deep neural networks in classification’, Theor. Found. Mach.e Learn., 2016, 25, pp. 4959.
    21. 21)
      • 21. Liu, W., Wen, Y., Yu, Z., et al: ‘Large-margin softmax loss for convolutional neural networks’. Int. Conf. Machine Learning, New York City, USA, June 2016, pp. 507516.
    22. 22)
      • 22. Xu, C., Lu, C., Liang, X., et al: ‘Multi-loss regularized deep neural network’, IEEE Trans. Circuits Syst. Video Technol., 2016, 26, (12), pp. 22732283.
    23. 23)
      • 23. Shaham, U., Lederman, R.R.: ‘Learning by coincidence: Siamese networks and common variable learning’, Pattern Recognit., 2018, 74, pp. 5263.
    24. 24)
      • 24. Chopra, S., Hadsell, R., LeCun, Y.: ‘Learning a similarity metric discriminatively, with application to face verification’. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, San Diego, CA, USA, June 2005, pp. 539546.
    25. 25)
      • 25. Schroff, F., Kalenichenko, D., Philbin, J.: ‘Facenet: a unified embedding for face recognition and clustering’. 2015 IEEE Conf. Computer Vision and Pattern Recognition, Boston, Massachusetts, USA, June 2015, pp. 815823.
    26. 26)
      • 26. Shore, J., Johnson, R.: ‘Properties of cross-entropy minimization’, IEEE Trans. Inf. Theory, 1981, 27, (4), pp. 472482.
    27. 27)
      • 27. Lin, M., Chen, Q., Yan, S.: ‘Network in network’, arXiv preprint arXiv:1312.4400, 2013.
    28. 28)
      • 28. Gu, J., Wang, Z., Kuen, J., et al: ‘Recent advances in convolutional neural networks’, Pattern Recognit., 2017, 77, pp. 354377.
    29. 29)
      • 29. Abadi, M., Agarwal, A., Barham, P., et al: ‘Tensorflow: large-scale machine learning on heterogeneous distributed systems’, arXiv preprint arXiv:1603.04467, 2016.
    30. 30)
      • 30. Krizhevsky, A., Hinton, G.: ‘Learning multiple layers of features from tiny images’. Master's thesis, Department of Computer Science, University of Toronto, 2009.
    31. 31)
      • 31. Netzer, Y., Wang, T., Coates, A., et al: ‘Reading digits in natural images with unsupervised feature learning’. Presented at the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain, December 2011.
    32. 32)
      • 32. Zhang, K., Guo, L., Gao, C., et al: ‘Pyramidal RoR for image classification’, Cluster Comput., 2017, pp. 111, doi: 10.1007/s10586-017-1443-x.
    33. 33)
      • 33. Han, D., Kim, J., Kim, J.: ‘Deep pyramidal residual networks’. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, Hawaii, July 2017, pp. 63076315.

Related content

This is a required field
Please enter a valid email address