Automatic image annotation by a loosely joint non-negative matrix factorisation

Roya Rad; Mansour Jamzad

Automatic image annotation by a loosely joint non-negative matrix factorisation

View Fulltext

Author(s): Roya Rad ¹ and Mansour Jamzad ¹
- Affiliations: 1: Department of Computer Engineering, Sharif University of Technology, Tehran, Iran
Source: Volume 9, Issue 6, December 2015, p. 806 – 813
DOI: 10.1049/iet-cvi.2014.0413 , Print ISSN 1751-9632, Online ISSN 1751-9640

Received 24/07/2014, Accepted 07/04/2015, Revised 16/03/2015, Published 01/12/2015

Nowadays, the number of digital images has increased so that the management of this volume of data needs an efficient system for browsing, categorising and searching. Automatic image annotation is designed for assigning tags to images for more accurate retrieval. Non-negative matrix factorisation (NMF) is a traditional machine learning technique for decomposing a matrix into a set of basis and coefficients under the non-negative constraints. In this study, the authors propose a two-step algorithm for designing an automatic image annotation system that employs the NMF framework for its first step and a variant of K-nearest neighbourhood as its second step. In the first step, a new multimodal NMF algorithm is proposed to extract the latent factors which reflect the content of images. This is done by jointly factorising the visual and textual data feature matrices so that they have close representation, although not necessarily the same. In the second step, after mapping images to the latent factors space a few tags are predicted for the new images based on a weighted average of similar data. They evaluated the performance of the proposed method and compared it to the state-of-the-art literature. Comparison results demonstrate the effectiveness and potential of the proposed method in image annotation applications.

References

1. 1)
  - 28. Kejun, H., Sidiropoulos, N.D., Swami, A.: ‘Non-negative matrix factorization revisited: uniqueness and algorithm for symmetric decomposition’, IEEE Trans. Signal Process., 2014, 62, (1), pp. 211–224 (doi: 10.1109/TSP.2013.2285514).
2. 2)
  - 8. Putthividhy, D., Attias, H.T., Nagarajan, S.S.: ‘Topic regression multi-modal latent Dirichlet allocation for image annotation’. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
3. 3)
  - 10. Zhang, R., Zhang, Z., Li, M., Ma, W.-Y., Zhang, H.-J.: ‘A probabilistic semantic model for image annotation and multimodal image retrieval’. Tenth IEEE Int. Conf. Computer Vision (ICCV), 2005.
4. 4)
  - 26. Acar, E., Gurdeniz, G., Rasmussen, M.A., Rago, D., Dragsted, L.O., Bro, R.: ‘Coupled matrix factorization with sparse factors to identify potential biomarkers in metabolomics’. IEEE 12th Int. Conf. Data Mining Workshops (ICDMW), 2012.
5. 5)
  - 4. Paatero, P., Tapper, U.: ‘Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values’, Environmetrics, 1994, 5, (2), pp. 111–126 (doi: 10.1002/env.3170050203).
6. 6)
  - 32. Manning, C.D., Raghavan, P., Schütze, H.: ‘Introduction to information retrieval’ (Cambridge university press Cambridge, 2008).
7. 7)
  - 9. Wang, C., Blei, D., Li, F.-F.: ‘Simultaneous image classification and annotation’. IEEE Computer Vision and Pattern Recognition (CVPR), 2009.
8. 8)
  - 15. Verma, Y., Jawahar, C.: ‘Exploring Svm for image annotation in presence of confusing labels’. Proc.Conf.24th British Machine Vision, 2013.
9. 9)
  - 30. Berry, M.W., Browne, M., Langville, A.N., Pauca, V.P., Plemmons, R.J.: ‘Algorithms and applications for approximate nonnegative matrix factorization’, Comput. Stat. Data Anal., 2007, 52, (1), pp. 155–173 (doi: 10.1016/j.csda.2006.11.006).
10. 10)
  - 40. Xiang, Y., Zhou, X., Chua, T.-S., Ngo, C.-W.: ‘A revisit of generative model for automatic image annotation using Markov random fields’. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2009.
11. 11)
  - 37. Oliva, A.: ‘Gist of the scene’, Neurobiol. Attention, 2005, 696, (64), pp. 251–258.
12. 12)
  - 17. Jinhui, T., Haojie, L., Qi, G.J., Chua, T.S.: ‘Image annotation by graph-based inference with integrated multiple/single instance representations’, IEEE Trans. Multimedia, 2010, 12, (2), pp. 131–141 (doi: 10.1109/TMM.2009.2037373).
13. 13)
  - 12. Caicedo, J.C., BenAbdallah, J., González, F.A., Nasraoui, O.: ‘Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization’, Neurocomputing, 2012, 76, (1), pp. 50–60 (doi: 10.1016/j.neucom.2011.04.037).
14. 14)
  - 7. Li, J., Wang, J.Z.: ‘Real-time computerized annotation of pictures’, IEEE Trans. Pattern Anal. Mach. Intell., 2008, 30, (6), pp. 985–1002 (doi: 10.1109/TPAMI.2007.70847).
15. 15)
  - 41. Lu, Z., Ip, H.H., Peng, Y.: ‘Contextual kernel and spectral methods for learning the semantics of images’, IEEE Trans. Image Process., 2011, 20, (6), pp. 1739–1750 (doi: 10.1109/TIP.2010.2103082).
16. 16)
  - 18. Savita, P., Patel, D., Sinhal, A.: ‘A neural network approach to improve the efficiency of image annotation’, Int. J. Eng. Res. Technol., 2013, 1, pp. 35–41.
17. 17)
  - 5. Lee, D.D., Seung, H.S.: ‘Learning the parts of objects by non-negative matrix factorization’, Nature, 1999, 401, (6755), pp. 788–791 (doi: 10.1038/44565).
18. 18)
  - 38. Chen, M., Zheng, A., Weinberger, K.: ‘Fast image tagging’. Proc.30th Int. Conf. Machine Learning, 2013.
19. 19)
  - 11. Lienhart, R., Romberg, S., Hörster, E.: ‘Multilayer plsa for multimodal image retrieval’. Proc. ACM Int. Conf. Image and Video Retrieval, 2009.
20. 20)
  - 25. Akata, Z., Thurau, C., Bauckhage, C.: ‘Non-negative matrix factorization in multimodality data for segmentation and label prediction’. 16th Computer Vision Winter Workshop, 2011.
21. 21)
  - 14. Kalayeh, M.M., Idrees, H., Shah, M.: ‘Nmf-Knn: image annotation using weighted multi-view non-negative matrix factorization’. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
22. 22)
  - 27. Yan, X., Guo, J., Liu, S., Cheng, X., Wang, Y.: ‘Learning topics in short texts by non-negative matrix factorization on term correlation matrix’. Proc. SIAM Int. Conf. Data Mining, 2013.
23. 23)
  - 6. Carneiro, G., Chan, A.B., Moreno, P.J., Vasconcelos, N.: ‘Supervised learning of semantic classes for image annotation and retrieval’, IEEE Trans. Pattern Anal. Mach. Intell., 2007, 29, (3), pp. 394–410 (doi: 10.1109/TPAMI.2007.61).
24. 24)
  - 3. Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: ‘Content-based image retrieval at the end of the early years’, IEEE Trans. Pattern Anal. Mach. Intell., 2000, 22, (12), pp. 1349–1380 (doi: 10.1109/34.895972).
25. 25)
  - 19. Guillaumin, M., Mensink, T., Verbeek, J., Schmid, C.: ‘Tagprop: discriminative metric learning in nearest neighbor models for image auto-annotation’. IEEE 12th Int. Conf. Computer Vision, 2009.
26. 26)
  - 35. Tsai, C.-F.: ‘Bag-of-words representation in image annotation: a review’. ISRN Artificial Intelligence, 2012.
27. 27)
  - 23. Caicedo, J.C., González, F.A.: ‘Multimodal fusion for image retrieval using matrix factorization’. Proc. Second ACM Int. Conf. Multimedia Retrieval, 2012.
28. 28)
  - 39. Lu, Z., Peng, Y.: ‘Image annotation by semantic sparse recoding of visual content’. Proc. 20th ACM Int. Conf. Multimedia, ACM, 2012.
29. 29)
  - 36. Lowe, D.G.: ‘Object recognition from local scale-invariant features’. Proc. Seventh IEEE Int. Conf. Computer Vision, 1999.
30. 30)
  - 22. Driesen, J.: ‘Discovering words in speech using matrix factorization’. PhD thesis, KU Leuven, ESAT, 2012.
31. 31)
  - 29. Caicedo, J.C., González, F.A.: ‘Online matrix factorization for multimodal image retrieval’. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Springer, 2012.
32. 32)
  - 33. Duygulu, P., Barnard, K., de Freitas, J.F., Forsyth, D.A.: ‘Object recognition as machine translation: learning a lexicon for a fixed image vocabulary’. Computer Vision, ECCV, Springer, 2002.
33. 33)
  - 20. Verma, Y., Jawahar, C.: ‘Image annotation using metric learning in semantic neighbourhoods’. Computer Vision (ECCV), 2012.
34. 34)
  - 31. Makadia, A., Pavlovic, V., Kumar, S.: ‘A new baseline for image annotation’. Computer Vision, ECCV, Springer, 2008.
35. 35)
  - 34. Von Ahn, L., Dabbish, L.: ‘Labeling images with a computer game’. Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM, 2004.
36. 36)
  - 13. Li, Z., Liu, J., Lu, H.: ‘Nonlinear matrix factorization with unified embedding for social tag relevance learning’, Neurocomputing, 2013, 105, pp. 38–44 (doi: 10.1016/j.neucom.2012.02.046).
37. 37)
  - 21. BenAbdallah, J., Caicedo, J.C., Gonzalez, F.A., Nasraoui, O.: ‘Multimodal image annotation using non-negative matrix factorization’. IEEE/WIC/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology (WI-IAT), 2010.
38. 38)
  - 1. ‘Instagram Statistics for 2014’, http://www.jennstrends.com/instagram-statistics-for-2014/, accessed September 2014.
39. 39)
  - 16. Richang, H., Meng, W., Yue, G., Dacheng, T., Xuelong, L., Xindong, W.: ‘Image annotation by multiple-instance learning with discriminative feature mapping and selection’, IEEE Trans. Cybern., 2014, 44, (5), pp. 669–680 (doi: 10.1109/TCYB.2013.2265601).
40. 40)
  - 2. Datta, R., Joshi, D., Li, J., Wang, J.Z.: ‘Image retrieval: ideas, influences, and trends of the new age’, ACM Comput. Surv. (CSUR), 2008, 40, (2), pp. 1–60 (doi: 10.1145/1348246.1348248).
41. 41)
  - 24. Eweiwi, A., Cheema, M.S., Bauckhage, C.: ‘Discriminative joint non-negative matrix factorization for human action classification’, Pattern Recogn., 2013, 8142, pp. 61–70.

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Automatic image annotation by a loosely joint non-negative matrix factorisation

References

Related content