Your browser does not support JavaScript!

access icon openaccess Compressing deep-quaternion neural networks with targeted regularisation

In recent years, hyper-complex deep networks (such as complex-valued and quaternion-valued neural networks – QVNNs) have received a renewed interest in the literature. They find applications in multiple fields, ranging from image reconstruction to 3D audio processing. Similar to their real-valued counterparts, quaternion neural networks require custom regularisation strategies to avoid overfitting. In addition, for many real-world applications and embedded implementations, there is the need of designing sufficiently compact networks, with few weights and neurons. However, the problem of regularising and/or sparsifying QVNNs has not been properly addressed in the literature as of now. In this study, the authors show how to address both problems by designing targeted regularisation strategies, which can minimise the number of connections and neurons of the network during training. To this end, they investigate two extensions of and structured regularisations to the quaternion domain. In the authors’ experimental evaluation, they show that these tailored strategies significantly outperform classical (real-valued) regularisation approaches, resulting in small networks especially suitable for low-power and real-time applications.


    1. 1)
    2. 2)
      • [11]. Comminiello, D., Lella, M., Scardapane, S., et al: ‘Quaternion convolutional neural networks for detection and localization of 3d sound events’. Proc. 2019 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, 2019, pp. 85338537.
    3. 3)
    4. 4)
    5. 5)
      • [2]. Zhu, X., Xu, Y., Xu, H., et al: ‘Quaternion convolutional neural networks’. Proc. 2018 European Conf. on Computer Vision (ECCV), Munich, Germany, 2018, pp. 631647.
    6. 6)
    7. 7)
    8. 8)
      • [16]. Wen, W., Wu, C., Wang, Y., et al: ‘Learning structured sparsity in deep neural networks’. Advances in Neural Information Processing Systems, Barcelona, Spain, 2016, pp. 20742082.
    9. 9)
      • [8]. Tay, Y., Zhang, A., Tuan, L.A., et al: ‘Lightweight and efficient neural natural language processing with quaternion networks’, arXiv preprint arXiv:190604393, 2019.
    10. 10)
    11. 11)
    12. 12)
      • [9]. Zhang, S., Tay, Y., Yao, L., et al: ‘Quaternion knowledge graph embedding’, arXiv preprint arXiv:190410281, 2019.
    13. 13)
      • [3]. Parcollet, T., Ravanelli, M., Morchid, M., et al: ‘Quaternion recurrent neural networks’, arXiv preprint arXiv:180604418, 2018.
    14. 14)
    15. 15)
    16. 16)
      • [21]. Gordon, A., Eban, E., Nachum, O., et al: ‘Morphnet: fast & simple resource-constrained structure learning of deep networks’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, Utah, USA, 2018, pp. 15861595.
    17. 17)
    18. 18)
      • [5]. Parcollet, T., Morchid, M., Linarès, G.: ‘Quaternion convolutional neural networks for heterogeneous image processing’. Proc. 2019 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, 2019, pp. 85148518.
    19. 19)
      • [22]. Ward, J.P.: ‘Quaternions and Cayley numbers: algebra and applications’ (Springer Science & Business Media, Berlin, Germany, 2012), vol. 403.
    20. 20)
      • [20]. Badeńska, A., Błaszczyk, Ł.: ‘Compressed sensing in the quaternion algebra’, arXiv preprint arXiv:170408202, 2017.
    21. 21)
    22. 22)
    23. 23)
    24. 24)
      • [24]. Ioffe, S., Szegedy, C.: ‘Batch normalization: accelerating deep network training by reducing internal covariate shift’, arXiv preprint arXiv:150203167, 2015.

Related content

This is a required field
Please enter a valid email address