Multi-modality-based Arabic sign language recognition

Multi-modality-based Arabic sign language recognition

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Computer Vision — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

With the increase in the number of deaf-mute people in the Arab world and the lack of Arabic sign language (ArSL) recognition benchmark data sets, there is a pressing need for publishing a large-volume and realistic ArSL data set. This study presents such a data set, which consists of 150 isolated ArSL signs. The data set is challenging due to the great similarity among hand shapes and motions in the collected signs. Along with the data set, a sign language recognition algorithm is presented. The authors’ proposed method consists of three major stages: hand segmentation, hand shape sequence and body motion description, and sign classification. The hand shape segmentation is based on the depth and position of the hand joints. Histograms of oriented gradients and principal component analysis are applied on the segmented hand shapes to obtain the hand shape sequence descriptor. The covariance of the three-dimensional joints of the upper half of the skeleton in addition to the hand states and face properties are adopted for motion sequence description. The canonical correlation analysis and random forest classifiers are used for classification. The achieved accuracy is 55.57% over 150 ArSL signs, which is considered promising.


    1. 1)
      • 1. Mohandes, M., Deriche, M., Liu, J.: ‘Image-based and sensor-based approaches to Arabic sign language recognition’, IEEE Trans. Hum.-Mach. Syst., 2014, 44, (4), pp. 551557.
    2. 2)
      • 2. Hussein, M. E., Torki, M., Gowayyed, M. A., et al: ‘Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations’. Proc. IJCAI, Beijing, China, August 2013, pp. 24662472.
    3. 3)
      • 3. Ong, S.C., Ranganath, S.: ‘Automatic sign language analysis: a survey and the future beyond lexical meaning’, IEEE Trans. Pattern Anal. Mach. Intell., 2005, 27, (6), pp. 873891.
    4. 4)
      • 4. Neidle, C., Thangali, A., Sclaroff, S.: ‘Challenges in development of the American sign language lexicon video dataset (ASLLVD) corpus’. Proc. 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, LREC, Istanbul, Turkey, 2012.
    5. 5)
      • 5. Dreuw, P., Neidle, C., Athitsos, V., et al: ‘Benchmark databases for video-based automatic sign language recognition’. Proc. Int. Conf. Language Resources and Evaluation (LREC), Marrakech, Morocco, 2008.
    6. 6)
      • 6. Wilbur, R., Kak, A.C.: ‘Purdue RVL-SLLL American sign language database’. Technical Report, School of Electrical and Computer Engineering, Purdue University, 2006.
    7. 7)
      • 7. Buehler, P., Everingham, M., Huttenlocher, D.P., et al: ‘Long term arm and hand tracking for continuous sign language TV broadcasts’. Proc. of the 19th British Machine Vision Conf., Leeds, UK, 2008, pp. 11051114.
    8. 8)
      • 8. Forster, J., Schmidt, C., Hoyoux, T., et al: ‘A large vocabulary sign language recognition and translation corpus’. Proc. Int. Conf. Language Resources and Evaluation (LREC), Istanbul, Turkey, May 2012, pp. 37853789.
    9. 9)
      • 9. Crasborn, O.A., Zwitserlood, I.E.P.: ‘The corpus NGT: an online corpus for professionals and laymen’. Proc. 3rd Workshop on the Representation and Processing of Sign Languages (LREC), Marrakech, Morocco, 2008, pp. 4449.
    10. 10)
      • 10. Bungeroth, J., Stein, D., Dreuw, P., et al: ‘The ATIS sign language corpus’. Proc. Int. Conf. Language Resources and Evaluation (LREC), Marrakech, Morocco, 2008.
    11. 11)
      • 11. Stefanov, K., Beskow, J.: ‘A kinect corpus of Swedish sign language signs’. Proc. of the 2013 Workshop on Multimodal Corpora: Beyond Audio and Video, Edinburgh, UK, 2013.
    12. 12)
      • 12. Almohimeed, A.: ‘Arabic text to Arabic sign language example-based translation system’. PhD thesis, University of Southampton, 2012.
    13. 13)
      • 13. Shanableh, T., Assaleh, K., Al-Rousan, M.: ‘Spatio-temporal feature-extraction techniques for isolated gesture recognition in Arabic sign language’, IEEE Trans. Syst., Man, Cybern., B (Cybernetics), 2007, 37, (3), pp. 641650.
    14. 14)
      • 14. Shohieb, S.M., Elminir, H.K., Riad, A.M.: ‘Signsworld atlas; a benchmark Arabic sign language database’, J. King Saud Univ.-Comput. Inf. Sci., 2015, 27, (1), pp. 6876.
    15. 15)
      • 15. Coogan, T., Awad, G., Han, J., et al: ‘Real time hand gesture recognition including hand segmentation and tracking’. J. Adv. Vis. Comput., 2006, 4291, pp. 495504.
    16. 16)
      • 16. Liwicki, S., Everingham, M.: ‘Automatic recognition of fingerspelled words in British sign language’. Proc. CVPR Workshops, Miami, FL, USA, June 2009, pp. 5057.
    17. 17)
      • 17. Shanableh, T., Assaleh, K.: ‘User-independent recognition of Arabic sign language for facilitating communication with the deaf community’, J. Digital Signal Process., 2011, 21, (4), pp. 535542.
    18. 18)
      • 18. Mohandes, M., Deriche, M., Johar, U., et al: ‘A signer-independent Arabic sign language recognition system using face detection, geometric features, and a hidden Markov model’, J. Comput. Electr. Eng., 2012, 38, (2), pp. 422433.
    19. 19)
      • 19. Kelly, D., Mc Donald, J., Markham, C.: ‘Continuous recognition of motion based gestures in sign language’. Proc. IEEE 12th Int. Conf. on Computer Vision Workshops, ICCV Workshops, Kyoto, Japan, September 2009, pp. 10731080.
    20. 20)
      • 20. Dong, C., Leu, M.C., Yin, Z.: ‘American sign language alphabet recognition using Microsoft kinect’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, June 2015, pp. 4452.
    21. 21)
      • 21. Hamed, A., Belal, N.A., Mahar, K.M.: ‘Arabic sign language alphabet recognition based on HOG-PCA using Microsoft kinect in complex backgrounds’. Proc. Int. Conf. on Advanced Computing (IACC), Bhimavaram, India, February 2016, pp. 451458.
    22. 22)
      • 22. ElSaadany, O.S., Abdelwahab, M.M.: ‘Real-time 2DHoG-2DPCA algorithm for hand gesture recognition’. Proc. Int. Conf. on Image Analysis and Processing, Naples, Italy, 2013, pp. 601610.
    23. 23)
      • 23. Ren, Z., Yuan, J., Meng, J., et al: ‘Robust part-based hand gesture recognition using kinect sensor’, IEEE Trans. Multimed., 2013, 15, (5), pp. 11101120.
    24. 24)
      • 24. Sako, S., Hatano, M., Kitamura, T.: ‘Real-time Japanese sign language recognition based on three phonological elements of sign’. Proc. Int. Conf. on Human-Computer Interaction, Toronto, Canada, July 2016, pp. 130136.
    25. 25)
      • 25. Zhang, C., Tian, Y., Huenerfauth, M.: ‘Multi-modality American sign language recognition’. Proc. Int. Conf. on Image Processing (ICIP), Phoenix, AZ, USA, September 2016, pp. 28812885.
    26. 26)
      • 26. Plouffe, G., Cretu, A.M.: ‘Static and dynamic hand gesture recognition in depth data using dynamic time warping’. IEEE Trans. Instrum. Meas., 2016, 65, (2), pp. 305316.
    27. 27)
      • 27. Dalal, N., Triggs, B.: ‘Histograms of oriented gradients for human detection’. Proc. Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA, June 2005, Vol. 1, pp. 886893.
    28. 28)
      • 28. Nicolaou, M.A., Panagakis, Y., Zafeiriou, S., et al: ‘Robust canonical correlation analysis: audio-visual fusion for learning continuous interest’. Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 15221526.
    29. 29)
      • 29. Escalera, S., Gonzàlez, J., Baró, X., et al: ‘Multi-modal gesture recognition challenge 2013: dataset and results’. Proc. The 15th ACM on Int. Conf. on multimodal interaction, Sydney, Australia, 2013, pp. 445452.
    30. 30)
      • 30. Ansari, Z.A., Harit, G.: ‘Nearest neighbour classification of Indian sign language gestures using kinect camera’, Sadhana, 2016, 41, (2), pp. 161182.

Related content

This is a required field
Please enter a valid email address