Multi-modality-based Arabic sign language recognition

Marwa Elpeltagy; Moataz Abdelwahab; Mohamed E. Hussein; Amin Shoukry; Asmaa Shoala; Moustafa Galal

Multi-modality-based Arabic sign language recognition

View Fulltext

Author(s): Marwa Elpeltagy¹ ; Moataz Abdelwahab¹ ; Mohamed E. Hussein^{2, 3} ; Amin Shoukry^{2, 4} ; Asmaa Shoala⁵ ; Moustafa Galal⁵
- Affiliations: 1: Department of Electronics and Communications Engineering , Egypt-Japan University of Science and Technology , Alexandria 21934 , Egypt ;
  2: Computer and Systems Engineering Department , Alexandria University , Alexandria 21544 , Egypt ;
  3: Information Sciences Institute , Arlington, VA 22203 , USA ;
  4: Department of Computer Science and Engineering , Egypt-Japan University of Science and Technology , Alexandria 21934 , Egypt ;
  5: ITAC Research Project , Alexandria 21934 , Egypt
Source: Volume 12, Issue 7, October 2018, p. 1031 – 1039
DOI: 10.1049/iet-cvi.2017.0598 , Print ISSN 1751-9632, Online ISSN 1751-9640

Received 26/11/2017, Accepted 12/05/2018, Revised 22/03/2018, Published 15/05/2018

With the increase in the number of deaf-mute people in the Arab world and the lack of Arabic sign language (ArSL) recognition benchmark data sets, there is a pressing need for publishing a large-volume and realistic ArSL data set. This study presents such a data set, which consists of 150 isolated ArSL signs. The data set is challenging due to the great similarity among hand shapes and motions in the collected signs. Along with the data set, a sign language recognition algorithm is presented. The authors’ proposed method consists of three major stages: hand segmentation, hand shape sequence and body motion description, and sign classification. The hand shape segmentation is based on the depth and position of the hand joints. Histograms of oriented gradients and principal component analysis are applied on the segmented hand shapes to obtain the hand shape sequence descriptor. The covariance of the three-dimensional joints of the upper half of the skeleton in addition to the hand states and face properties are adopted for motion sequence description. The canonical correlation analysis and random forest classifiers are used for classification. The achieved accuracy is 55.57% over 150 ArSL signs, which is considered promising.

References

1. 1)
  - 12. Almohimeed, A.: ‘Arabic text to Arabic sign language example-based translation system’. PhD thesis, University of Southampton, 2012.
2. 2)
  - 8. Forster, J., Schmidt, C., Hoyoux, T., et al: ‘A large vocabulary sign language recognition and translation corpus’. Proc. Int. Conf. Language Resources and Evaluation (LREC), Istanbul, Turkey, May 2012, pp. 3785–3789.
3. 3)
  - 23. Ren, Z., Yuan, J., Meng, J., et al: ‘Robust part-based hand gesture recognition using kinect sensor’, IEEE Trans. Multimed., 2013, 15, (5), pp. 1110–1120.
4. 4)
  - 24. Sako, S., Hatano, M., Kitamura, T.: ‘Real-time Japanese sign language recognition based on three phonological elements of sign’. Proc. Int. Conf. on Human-Computer Interaction, Toronto, Canada, July 2016, pp. 130–136.
5. 5)
  - 27. Dalal, N., Triggs, B.: ‘Histograms of oriented gradients for human detection’. Proc. Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA, June 2005, Vol. 1, pp. 886–893.
6. 6)
  - 16. Liwicki, S., Everingham, M.: ‘Automatic recognition of fingerspelled words in British sign language’. Proc. CVPR Workshops, Miami, FL, USA, June 2009, pp. 50–57.
7. 7)
  - 5. Dreuw, P., Neidle, C., Athitsos, V., et al: ‘Benchmark databases for video-based automatic sign language recognition’. Proc. Int. Conf. Language Resources and Evaluation (LREC), Marrakech, Morocco, 2008.
8. 8)
  - 22. ElSaadany, O.S., Abdelwahab, M.M.: ‘Real-time 2DHoG-2DPCA algorithm for hand gesture recognition’. Proc. Int. Conf. on Image Analysis and Processing, Naples, Italy, 2013, pp. 601–610.
9. 9)
  - 29. Escalera, S., Gonzàlez, J., Baró, X., et al: ‘Multi-modal gesture recognition challenge 2013: dataset and results’. Proc. The 15th ACM on Int. Conf. on multimodal interaction, Sydney, Australia, 2013, pp. 445–452.
10. 10)
  - 17. Shanableh, T., Assaleh, K.: ‘User-independent recognition of Arabic sign language for facilitating communication with the deaf community’, J. Digital Signal Process., 2011, 21, (4), pp. 535–542.
11. 11)
  - 11. Stefanov, K., Beskow, J.: ‘A kinect corpus of Swedish sign language signs’. Proc. of the 2013 Workshop on Multimodal Corpora: Beyond Audio and Video, Edinburgh, UK, 2013.
12. 12)
  - 30. Ansari, Z.A., Harit, G.: ‘Nearest neighbour classification of Indian sign language gestures using kinect camera’, Sadhana, 2016, 41, (2), pp. 161–182.
13. 13)
  - 18. Mohandes, M., Deriche, M., Johar, U., et al: ‘A signer-independent Arabic sign language recognition system using face detection, geometric features, and a hidden Markov model’, J. Comput. Electr. Eng., 2012, 38, (2), pp. 422–433.
14. 14)
  - 10. Bungeroth, J., Stein, D., Dreuw, P., et al: ‘The ATIS sign language corpus’. Proc. Int. Conf. Language Resources and Evaluation (LREC), Marrakech, Morocco, 2008.
15. 15)
  - 14. Shohieb, S.M., Elminir, H.K., Riad, A.M.: ‘Signsworld atlas; a benchmark Arabic sign language database’, J. King Saud Univ.-Comput. Inf. Sci., 2015, 27, (1), pp. 68–76.
16. 16)
  - 2. Hussein, M. E., Torki, M., Gowayyed, M. A., et al: ‘Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations’. Proc. IJCAI, Beijing, China, August 2013, pp. 2466–2472.
17. 17)
  - 4. Neidle, C., Thangali, A., Sclaroff, S.: ‘Challenges in development of the American sign language lexicon video dataset (ASLLVD) corpus’. Proc. 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, LREC, Istanbul, Turkey, 2012.
18. 18)
  - 28. Nicolaou, M.A., Panagakis, Y., Zafeiriou, S., et al: ‘Robust canonical correlation analysis: audio-visual fusion for learning continuous interest’. Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 1522–1526.
19. 19)
  - 19. Kelly, D., Mc Donald, J., Markham, C.: ‘Continuous recognition of motion based gestures in sign language’. Proc. IEEE 12th Int. Conf. on Computer Vision Workshops, ICCV Workshops, Kyoto, Japan, September 2009, pp. 1073–1080.
20. 20)
  - 3. Ong, S.C., Ranganath, S.: ‘Automatic sign language analysis: a survey and the future beyond lexical meaning’, IEEE Trans. Pattern Anal. Mach. Intell., 2005, 27, (6), pp. 873–891.
21. 21)
  - 9. Crasborn, O.A., Zwitserlood, I.E.P.: ‘The corpus NGT: an online corpus for professionals and laymen’. Proc. 3rd Workshop on the Representation and Processing of Sign Languages (LREC), Marrakech, Morocco, 2008, pp. 44–49.
22. 22)
  - 26. Plouffe, G., Cretu, A.M.: ‘Static and dynamic hand gesture recognition in depth data using dynamic time warping’. IEEE Trans. Instrum. Meas., 2016, 65, (2), pp. 305–316.
23. 23)
  - 20. Dong, C., Leu, M.C., Yin, Z.: ‘American sign language alphabet recognition using Microsoft kinect’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, June 2015, pp. 44–52.
24. 24)
  - 21. Hamed, A., Belal, N.A., Mahar, K.M.: ‘Arabic sign language alphabet recognition based on HOG-PCA using Microsoft kinect in complex backgrounds’. Proc. Int. Conf. on Advanced Computing (IACC), Bhimavaram, India, February 2016, pp. 451–458.
25. 25)
  - 6. Wilbur, R., Kak, A.C.: ‘Purdue RVL-SLLL American sign language database’. Technical Report, School of Electrical and Computer Engineering, Purdue University, 2006.
26. 26)
  - 13. Shanableh, T., Assaleh, K., Al-Rousan, M.: ‘Spatio-temporal feature-extraction techniques for isolated gesture recognition in Arabic sign language’, IEEE Trans. Syst., Man, Cybern., B (Cybernetics), 2007, 37, (3), pp. 641–650.
27. 27)
  - 1. Mohandes, M., Deriche, M., Liu, J.: ‘Image-based and sensor-based approaches to Arabic sign language recognition’, IEEE Trans. Hum.-Mach. Syst., 2014, 44, (4), pp. 551–557.
28. 28)
  - 25. Zhang, C., Tian, Y., Huenerfauth, M.: ‘Multi-modality American sign language recognition’. Proc. Int. Conf. on Image Processing (ICIP), Phoenix, AZ, USA, September 2016, pp. 2881–2885.
29. 29)
  - 15. Coogan, T., Awad, G., Han, J., et al: ‘Real time hand gesture recognition including hand segmentation and tracking’. J. Adv. Vis. Comput., 2006, 4291, pp. 495–504.
30. 30)
  - 7. Buehler, P., Everingham, M., Huttenlocher, D.P., et al: ‘Long term arm and hand tracking for continuous sign language TV broadcasts’. Proc. of the 19th British Machine Vision Conf., Leeds, UK, 2008, pp. 1105–1114.

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Multi-modality-based Arabic sign language recognition

References

Related content