http://iet.metastore.ingenta.com
1887

Joint optimisation convex-negative matrix factorisation for multi-modal image collection summarisation based on images and tags

Joint optimisation convex-negative matrix factorisation for multi-modal image collection summarisation based on images and tags

For access to this article, please select a purchase option:

Buy eFirst article PDF
£12.50
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
— Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Image collection summarisation aims to represent a large-scale multi-modal collection with a small subset of images and tags, helping navigate a large image dataset. Most extant methods leverage the contributions of text-to-visual summaries, ignoring the visual contribution to the textual topic. When the tags are weakly labelled, the textual topic cannot accurately reflect the visual summary. To solve this, the authors propose a novel model, joint optimisation of convex non-negative matrix factorisation, which incorporates images and tags in a beneficial way. The objective function contains visual and textual error functions, sharing the same indicator matrix, connecting different modal relations. Then, they propose an iterative algorithm to optimise the proposed model. Finally, they explore the effects of different visual feature representations (e.g. bag-of-words and deep learning) on multi-modal collection summary. Our proposed method is then compared with state-of-the-art algorithms using two multi-modal datasets (i.e. MIRFlickr and NUS-WIDE-SCENE). Experimental results demonstrate the effectiveness of their proposed approach.

References

    1. 1)
      • F. Sadeghi , J.R. Tena , A. Farhadi .
        1. Sadeghi, F., Tena, J.R., Farhadi, A., et al: ‘Learning to select and order vacation photographs’. Applications of Computer Vision IEEE, Waikoloa, HI, USA, January 2015, pp. 510517.
        . Applications of Computer Vision IEEE , 510 - 517
    2. 2)
      • J. Wang , L. Jia , X.-S. Hua .
        2. Wang, J., Jia, L., Hua, X.-S.: ‘Interactive browsing via diversified visual summarization for image search results’, Multimedia Syst., 2011, 17, (5), pp. 379391.
        . Multimedia Syst. , 5 , 379 - 391
    3. 3)
      • D.G. Lowe .
        3. Lowe, D.G.: ‘Distinctive image features from scale invariant keypoints’, Int. J. Comput. Vis., 2004, 60, (2), pp. 91110.
        . Int. J. Comput. Vis. , 2 , 91 - 110
    4. 4)
      • G. Csurka , C.R. Dance , L. Fan .
        4. Csurka, G., Dance, C.R., Fan, L., et al: ‘Visual categorization with bags of keypoints’, Workshop Stat. Learn. Comput. Vis. ECCV, 2004, 44, (247), pp. 122.
        . Workshop Stat. Learn. Comput. Vis. ECCV , 247 , 1 - 22
    5. 5)
      • Y. Hadi , F. Essannouni , R.O.H. Thami .
        5. Hadi, Y., Essannouni, F., Thami, R.O.H.: ‘Video summarization by K-medoid clustering’. Proc. ACM Symp. Applied Computing, Dijon, France, April 2006, pp. 14001401.
        . Proc. ACM Symp. Applied Computing , 1400 - 1401
    6. 6)
      • P. Clough , H. Joho , M. Sanderson .
        6. Clough, P., Joho, H., Sanderson, M.: ‘Automatically organising images using concept hierarchies’. Proc. Multimedia Workshop running at ACM SIGIR Conf., January 2005, pp. 3339.
        . Proc. Multimedia Workshop running at ACM SIGIR Conf. , 33 - 39
    7. 7)
      • P. Schmitz .
        7. Schmitz, P.: ‘Inducing ontology from Flickr tags’. Collaborative Web Tagging Workshop at WWW 2006, Edinburgh, Scotland, May 2006.
        . Collaborative Web Tagging Workshop at WWW 2006
    8. 8)
      • A. Jaffe , M. Naaman , T. Tassa .
        8. Jaffe, A., Naaman, M., Tassa, T., et al: ‘Generating summaries and visualization for large collections of geo-referenced photographs’. Proc. Eighth ACM Int. Workshop on Multimedia Information Retrieval, Santa Barbara, CA, USA, October 2006, pp. 8998.
        . Proc. Eighth ACM Int. Workshop on Multimedia Information Retrieval , 89 - 98
    9. 9)
      • C.H. Li , C.Y. Chiu , C.R. Huang .
        9. Li, C.H., Chiu, C.Y., Huang, C.R., et al: ‘Image content clustering and summarization for photo collections’. IEEE Int. Conf. Multimedia and Expo, Toronto, ON, Canada, July 2006, pp. 10331036.
        . IEEE Int. Conf. Multimedia and Expo , 1033 - 1036
    10. 10)
      • I. Simon , N. Snavely , S.M. Seitz .
        10. Simon, I., Snavely, N., Seitz, S.M.: ‘Scene summarization for online image collections’. IEEE 11th Int. Conf. Computer Vision, Rio de Janeiro, Brazil, October 2007, pp. 18.
        . IEEE 11th Int. Conf. Computer Vision , 1 - 8
    11. 11)
      • C.L. Yang , J.L. Shen , J.Y. Peng .
        11. Yang, C.L., Shen, J.L., Peng, J.Y., et al: ‘Image collection summarization via dictionary learning for sparse representation’, Pattern Recognit., 2013, 46, (3), pp. 948961.
        . Pattern Recognit. , 3 , 948 - 961
    12. 12)
      • Q. Wang , J. Wan , Y. Yuan .
        12. Wang, Q., Wan, J., Yuan, Y.: ‘Locality constraint distance metric learning for traffic congestion detection’, Pattern Recognit., 2018, 75, pp. 272281.
        . Pattern Recognit. , 272 - 281
    13. 13)
      • H. Fang , W. Lu , F. Wu .
        13. Fang, H., Lu, W., Wu, F., et al: ‘Topic aspect-oriented summarization via group selection’, Neurocomputing, 2015, 149, pp. 16131619.
        . Neurocomputing , 1613 - 1619
    14. 14)
      • A. Krizhevsky , I. Sutskever , G.E. Hinton .
        14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ‘ImageNet classification with deep convolutional neural networks’. Int. Conf. Neural Information Processing Systems, Lake Tahoe, NV, December 2012, pp. 10971105.
        . Int. Conf. Neural Information Processing Systems , 1097 - 1105
    15. 15)
      • C. Szegedy , W. Liu , Y. Jia .
        15. Szegedy, C., Liu, W., Jia, Y., et al: ‘Going deeper with convolutions’. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015.
        . Proc. IEEE Conf. Computer Vision and Pattern Recognition
    16. 16)
      • K. He , X. Zhang , S. Ren .
        16. He, K., Zhang, X., Ren, S., et al: ‘Deep residual learning for image recognition’, arXiv:1512.03385 [cs], 2015.
        .
    17. 17)
      • K. Simonyan , A. Zisserman .
        17. Simonyan, K., Zisserman, A.: ‘Very deep convolutional networks for large-scale image recognition’, arXiv preprint arXiv:1409.1556, 2014.
        .
    18. 18)
      • Q. Wang , J. Gao , Y. Yuan .
        18. Wang, Q., Gao, J., Yuan, Y.: ‘Embedding structured contour and location prior in siamesed fully convolutional networks for road detection’, IEEE Trans. Intell. Transp. Syst., 2018, 19, (1), pp. 230241.
        . IEEE Trans. Intell. Transp. Syst. , 1 , 230 - 241
    19. 19)
      • H. Xu , J. Wang , X.-S. Hua .
        19. Xu, H., Wang, J., Hua, X.-S., et al: ‘Hybrid image summarization’. Proc. 19th ACM Int. Conf. Multimedia, Scottsdale, AZ, USA, December 2011, pp. 12171220.
        . Proc. 19th ACM Int. Conf. Multimedia , 1217 - 1220
    20. 20)
      • M.X. Li , C.X. Zhao , J.H. Tang .
        20. Li, M.X., Zhao, C.X., Tang, J.H.: ‘Hybrid image summarization by hypergraph partition’, Neurocomputing, 2013, 119, pp. 4148.
        . Neurocomputing , 41 - 48
    21. 21)
      • B. Zhou , V. Jagadeesh , R.: Piramuthu .
        21. Zhou, B., Jagadeesh, V., Piramuthu, R.:ConceptLearner: discovering visual concepts from weakly labeled image collections’. Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, MA, USA, June 2015, pp. 14921500.
        . Proc. IEEE Conf. Computer Vision and Pattern Recognition , 1492 - 1500
    22. 22)
      • J.E. Camargo , F.A. GonzÃlez .
        22. Camargo, J.E., GonzÃlez, F.A.: ‘Multimodal latent topic analysis for image collection summarization’, Inf. Sci., 2016, 328, pp. 270287.
        . Inf. Sci. , 270 - 287
    23. 23)
      • C.H.Q. Ding , T. Li , M.I. Jordan .
        23. Ding, C.H.Q., Li, T., Jordan, M.I.: ‘Convex and semi-nonnegative matrix factorizations’, IEEE Trans. Pattern Anal. Mach. Intell., 2010, 32, (1), pp. 4555.
        . IEEE Trans. Pattern Anal. Mach. Intell. , 1 , 45 - 55
    24. 24)
      • M.J. Huiskes , M.S. Lew .
        24. Huiskes, M.J., Lew, M.S.: ‘The MirFlickr retrieval evaluation’. Proc. First ACM Int. Conf. Multimedia Information Retrieval, Vancouver, British Columbia, Canada, October 2008, pp. 3943.
        . Proc. First ACM Int. Conf. Multimedia Information Retrieval , 39 - 43
    25. 25)
      • T.-S. Chua , J. Tang , R. Hong .
        25. Chua, T.-S., Tang, J., Hong, R., et al: ‘NUS-WIDE: a real-world web image database from National University of Singapore’. Proc. ACM Int. Conf. Image and Video Retrieval, Santorini, Fira, Greece, July 2009, pp. 4857.
        . Proc. ACM Int. Conf. Image and Video Retrieval , 48 - 57
    26. 26)
      • X. Zhu , A.B. Goldberg , J. Van Gael .
        26. Zhu, X., Goldberg, A.B., Van Gael, J., et al: ‘Improving diversity in ranking using absorbing random walks’, Phys. Lab., Univ. Wash., 2007, pp. 97104.
        . Phys. Lab., Univ. Wash. , 97 - 104
    27. 27)
      • Y. Jia , E. Shelhamer , J. Donahue .
        27. Jia, Y., Shelhamer, E., Donahue, J., et alCaffe: convolutional architecture for fast feature embedding’. Proc. ACM Int. Conf. Multimedia, Orlando, FL, USA, November 2014, pp. 675678.
        . Proc. ACM Int. Conf. Multimedia , 675 - 678
    28. 28)
      • C.-J. Lin .
        28. Lin, C.-J.: ‘Projected gradient methods for nonnegative matrix factorization’, Neural Comput., 2007, 19, (10), pp. 27562779.
        . Neural Comput. , 10 , 2756 - 2779
    29. 29)
      • G. Salton , A. Wong , C.S. Yang .
        29. Salton, G., Wong, A., Yang, C.S.: ‘A vector space model for automatic indexing’, Commun. ACM, 1975, 18, (11), pp. 613620.
        . Commun. ACM , 11 , 613 - 620
    30. 30)
      • H. Yu , Z.-H. Deng , Y. Yang .
        30. Yu, H., Deng, Z.-H., Yang, Y., et al: ‘A joint optimization model for image summarization based on image content and tags’. 28th AAAI Conf. Artificial Intelligence, Québec City, Québec, Canada, July 2014, pp. 215221.
        . 28th AAAI Conf. Artificial Intelligence , 215 - 221
    31. 31)
      • C.L. Yang , J.Y. Peng , J.P. Fan .
        31. Yang, C.L., Peng, J.Y., Fan, J.P.: ‘Image collection summarization via dictionary learning for sparse representation’. IEEE Conf. Computer Vision and Pattern Recognition, Providence, RI, USA, June 2012, pp. 11221129.
        . IEEE Conf. Computer Vision and Pattern Recognition , 1122 - 1129
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2017.0568
Loading

Related content

content/journals/10.1049/iet-cvi.2017.0568
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address