Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free Wise optimisation: deep image embedding by informative pair weighting and ranked list learning

  • Author(s): Lili Fan 1, 2 ; Hongwei Zhao 1, 2 ; Haoyu Zhao 3
    • View affiliations
  • Source: Volume 14, Issue 15, 15 December 2020, p. 3802 – 3811
    DOI:  10.1049/iet-ipr.2020.0454 , Print ISSN 1751-9659, Online ISSN 1751-9667
© The Institution of Engineering and Technology
Received 28/03/2020, Accepted 02/11/2020, Revised 14/10/2020, Published 04/12/2020

Deep image embedding learns how to map images onto feature vectors. Image retrieval performance is often used to evaluate embedding quality. In this study, the authors proposed a wise deep image embedding optimisation (WDIEO) algorithm based on informative pair weighting and ranked list learning (IPWRLL) for network optimisation of fine-grained image retrieval. First, a hard sample mining method Top-k is proposed to select positive and negative samples. Then, for the selected query sample, a ranking list is obtained by comparing the similarity between samples in the data set and the query sample, and the sample is labelled according to the similarity. Finally, for positive samples, two optimisation rules with different functions are used, while ensuring two key issues of instance weighting and intra-class data distribution. For negative samples, different from the widely adopted methods based on the weight of sample information, the authors’ algorithm's weights are set according to the ranking list, which keeps the inter-class data distribution and the optimisation direction consistent with the loss reduction direction. The WDIEO-IPWRLL model is an end-to-end optimisation that can share parameters in the testing process. Experiments show that their proposed model achieves the state-of-the-art performance on the benchmark data set.

Inspec keywords: learning (artificial intelligence); image retrieval; optimisation; feature extraction; image representation; data mining

Other keywords: network optimisation; instance weighting; sample information; positive samples; image retrieval performance; embedding quality; negative samples; selected query sample; optimisation direction; hard sample mining method Top-k; fine-grained image retrieval; map images; informative pair weighting; end-to-end optimisation; optimisation rules; wise optimisation; ranking list; ranked list learning; wise deep image embedding optimisation algorithm

Subjects: Computer vision and image processing techniques; Information retrieval techniques; Data mining; Optical, image and video signal processing; Optimisation techniques; Optimisation techniques; Data handling techniques

References

    1. 1)
      • 33. Weinberger, K.Q., Saul, L.: ‘Distance metric learning for large margin nearest neighbor classification’. Proc. of the Neural Information Processing Systems, Vancouver, BC, Canada, February 2009, pp. 207–244.
    2. 2)
      • 15. Oh Song, H., Xiang, Y., Jegelka, S., et al: ‘Deep metric learning via lifted structured feature embedding’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, June 2016, pp. 4004–4012.
    3. 3)
      • 28. Ustinova, E., Lempitsky, V.: ‘Learning deep embeddings with histogram loss’. Proc. of the Advances in Neural Information Processing Systems, Barcelona, Spain, December 2016, pp. 4170–4178.
    4. 4)
      • 29. Fan, L., Zhao, H., Zhao, H., et al: ‘Image retrieval based on learning to rank and multiple loss’, Int. J. Geo-Inf., 2019, 8, (9), pp. 393–415.
    5. 5)
      • 1. Yuan, Y., Yang, K., Zhang, C.: ‘Hard-aware deeply cascaded embedding’. Proc. of the IEEE Int. Conf. on Computer Vision, Honolulu, USA, October 2017, pp. 814–823.
    6. 6)
      • 25. Lin, T.Y., Goyal, P., Girshick, R., et al: ‘Focal loss for dense object detection’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 2017, pp. 2999–3007.
    7. 7)
      • 17. Yi, D., Lei, Z., Li, Z.: ‘Deep metric learning for practical person re-identification’. Proc. of the Int. Conf. on Pattern Recognition, Stockholm, Sweden, August 2014, pp. 34–39.
    8. 8)
      • 34. Movshovitz-Attias, Y., Toshev, A., Leung, T.K., et al: ‘No fuss distance metric learning using proxies’. Proc. of the IEEE Int. Conf. on Computer Vision, Honolulu, HI, USA, July 2017, pp. 360–368.
    9. 9)
      • 18. Hoffer, E., Ailon, N.: ‘Deep metric learning using triplet network’. Int. Workshop on Similarity-Based Pattern Recognition, Copenhagen, Denmark, 2015, vol. 10, no. 25, pp. 84–92.
    10. 10)
      • 22. Liu, Z., Luo, P., Qiu, S., et al: ‘DeepFashion: powering robust clothes recognition and retrieval with rich annotations’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, Nevada, June 2016, pp. 1096–1104.
    11. 11)
      • 8. Wan, J., Wang, D., Hoi, S.C.H., et al: ‘Deep learning for content-based image retrieval: a comprehensive study’. Proc. of ACM Int. Conf. on Multimedia, Orlando, USA, November 2014, pp. 157–166.
    12. 12)
      • 27. Chang, H.-S., Learned-Miller, E., McCallum, A.: ‘Active bias: training more accurate neural networks by emphasizing high variance samples’. Proc. of the Neural Information Processing Systems, Long Beach, USA, December 2017.
    13. 13)
      • 31. Russakovsky, O., Deng, J., Su, H., et al: ‘Imagenet large scale visual recognition challenge’, Int. J. Comput. Vis., 2015, 115, (3), pp. 211–252.
    14. 14)
      • 14. Huang, C., Loy, C.C., Tang, X.: ‘Local similarity-aware deep feature embedding’. Proc. of the Conf. on Neural Information Processing Systems, Barcelona, Spain, October 2016.
    15. 15)
      • 3. Schroff, F., Kalenichenko, D., Philbin, J.: ‘FaceNet: a unified embedding for face recognition and clustering’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Boston, MA, USA, June 2015, pp. 815–823.
    16. 16)
      • 30. Ioffe, S., Szegedy, C.: ‘Batch normalization: accelerating deep network training by reducing internal covariate shift’. Proc. of Int. Conf. on Machine Learning, Lille, France, July 2015, pp. 448–456.
    17. 17)
      • 26. Jiang, L., Zhou, Z., Leung, T., et al: ‘MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels’. Proc. of the Int. Conf. on Machine Learning, Stockholmsmässan, Sweden, July 2018, pp. 2304–2313.
    18. 18)
      • 5. Oh Song, H., Jegelka, S., Rathod, V., et al: ‘Deep metric learning via facility location’. Proc. of the IEEE Int. Conf. on Computer Vision, Honolulu, Hawaii, Utah, October 2017, pp. 5382–5390.
    19. 19)
      • 12. Ge, W.P: ‘Deep metric learning with hierarchical triplet loss’. Proc. of the European Conf. on Computer Vision, Munich, Germany, September 2018, pp. 269–285.
    20. 20)
      • 24. Lu, J., Xu, C., Zhang, W., et al: ‘Sampling wisely: deep image embedding by top-K precision optimization’. Proc. of the Int. Conf. on Computer Vision, Seoul, Korea, October 2019, pp. 7961–7970.
    21. 21)
      • 21. Krause, J., Stark, M., Deng, J., et al: ‘3D object representations for fine-grained categorization’. Proc. of the IEEE Int. Conf. on Computer Vision Workshops, Darling Harbour, Sydney, December 2013, pp. 554–561.
    22. 22)
      • 2. Wang, H., Wang, Y., Zhou, Z., et al: ‘CosFace: large margin cosine loss for deep face recognition’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake, Utah, June 2018, pp. 5265–5274.
    23. 23)
      • 35. Opitz, M., Waltner, G., Possegger, H., et al: ‘Deep metric learning with bier: boosting independent embeddings robustly’, IEEE Trans. Pattern Anal. Mach. Intell., 2018, 42, (2), pp. 276–292.
    24. 24)
      • 19. Wang, X., Hua, Y., Kodirov, E., et al: ‘Ranked list loss for deep metric learning’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Long Beach, CA, June 2019, pp. 5207–5216.
    25. 25)
      • 36. Kim, W., Goyal, B., Chawla, K., et al: ‘Attention-based ensemble for deep metric learning’. Proc. of the European Conf. on Computer Vision, Munich, Germany, September 2018, pp. 736–751.
    26. 26)
      • 6. Liang, R.-Z., Shi, L., Wang, H., et al: ‘Optimizing top precision performance measure of content-based image retrieval by learning similarity function’. Proc. of the IEEE Int. Conf. on Pattern Recognition, Cancún, Mexico, December 2016, pp. 2954–2958.
    27. 27)
      • 10. Wang, X., Han, X., Huang, W., et al: ‘Multi-similarity loss with general pair weighting for deep metric learning’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, CA, USA, June 2019, pp. 5022–5030.
    28. 28)
      • 7. Leal-Taixé, L., Canton-Ferrer, C., Schindler, K.: ‘Learning by tracking: Siamese CNN for robust target association’. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, June 2016, pp. 33–40.
    29. 29)
      • 32. Sohn, K.: ‘Improved deep metric learning with multi-class N-pair loss objective’. Proc. of Advances in Neural Information Processing Systems, Barcelona, Spain, December 2016, pp. 1857–1865.
    30. 30)
      • 20. Wah, C., Branson, S., Welinder, P., et al: ‘The Caltech-UCSD birds-200-2011 dataset’, California Institute of Technology, 2011, vol. 7.
    31. 31)
      • 16. Yang, X., Zhou, P., Wang, M.W.: ‘Person reidentification via structural deep metric learning’, IEEE Trans. Neural Netw. Learn. Syst., 2018, 8, pp. 1–12.
    32. 32)
      • 13. Law, M.T., Thome, N., Cord, M.: ‘Quadruplet-wise image similarity learning’. Proc. of the IEEE Int. Conf. on Computer Vision, Sydney, Australia, December 2013, pp. 249–256.
    33. 33)
      • 23. Harwood, B., Kumar, B., Carneiro, G., et al: ‘Smart mining for deep metric learning’. Proc. of the IEEE Int. Conf. on Computer Vision, Venice, Italy, April 2017, pp. 2821–2829.
    34. 34)
      • 11. Chopra, S., Hadsell, R., LeCun, Y.: ‘Learning a similarity metric discriminatively, with application to face verification’. Proc. of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, San Diego, CA, USA, June 2005, pp. 539–546.
    35. 35)
      • 9. Ning, Q., Zhu, J., Zhong, Z., et al: ‘Scalable image retrieval by sparse product quantization’, IEEE Trans. Multimed., 2016, 19, (3), pp. 586–597.
    36. 36)
      • 4. Guo, X., Gao, L., Liu, X., et al: ‘Improved deep embedded clustering with local structure preservation’. Proc. Int. Joint Conf. Artificial Intelligence, Melbourne, Australia, August 2017, pp. 1753–1759.
    37. 37)
      • 37. Wu, C.-Y., Manmatha, R., Smola, A.J., et al: ‘Sampling matters in deep embedding learning’. Proc. of the IEEE Int. Conf. on Computer Vision, Venice, Italy, October 2017, pp. 2840–2848.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-ipr.2020.0454
Loading

Related content

content/journals/10.1049/iet-ipr.2020.0454
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address