This is an open access article published by the IET under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/)
This study proposes an automatic sign language translator, which is developed as assistive technology to help the hearing/speech impaired communities to communicate with the rest of the world. The system architecture, which includes feature extraction and recognition stages is described in detail. The signs are classified into two types: static and dynamic. Various types of sign features are presented and analysed. Recognition stage considers the hidden Markov model and segmentation signature. Real-time implementation of the system with the use of Windows7 and LINUX Fedora 16 operating systems with VMware workstation is presented in detail. The system has been successfully tested on Malaysian sign language.
References
-
-
1)
-
8. Chang, F., Chen, C.-J., Lu, C.-J.: ‘A linear-time component-labeling algorithm using contour tracing technique’, Comput. Vis. Image Underst., 2004, 93, (2), pp. 206–220 (doi: 10.1016/j.cviu.2003.09.002).
-
2)
-
1. Pavlovic, V.I., Sharma, R., Huang, T.S.: ‘Visual interpretation of hand gestures for human computer interaction: a review’, IEEE Trans. Pattern Anal. Mach. Intell., 1999, 19, (7), pp. 677–695 (doi: 10.1109/34.598226).
-
3)
-
17. Segouat, J.: ‘A study of sign language coarticulation’, Spec. Interest Group Accessible Comput. (SIGACCESS), 2009, 2009, (93), pp. 31–38 (doi: 10.1145/1531930.1531935).
-
4)
-
18. San-Segundo, R., Pardo, J.M., Ferreiros, J., et al: ‘Spoken Spanish generation from sign language’, Interact. Comput., 2010, 22, (2), pp. 123–139 (doi: 10.1016/j.intcom.2009.11.011).
-
5)
-
35. Li, H., Greenspan, M.: ‘Model-based segmentation and recognition of dynamic gestures in continuous video streams’, Pattern Recognit., 2011, 44, (8), pp. 1614–1628 (doi: 10.1016/j.patcog.2010.12.014).
-
6)
-
22. Viblis, M.K., Kyriakopoulos, K.J.: ‘Gesture recognition: the gesture segmentation problem’, J. Intell. Robot. Syst., 2000, 28, pp. 151–158 (doi: 10.1023/A:1008101200733).
-
7)
-
4. Alon, J., Athitsos, V., Yuan, Q., Sclaroff, S.: ‘A unified framework for gesture recognition and spatiotemporal gesture segmentation’, Trans. Pattern Anal. Mach. Intell., 2009, 31, (9), pp. 1685–1699 (doi: 10.1109/TPAMI.2008.203).
-
8)
-
26. Kong, W.W., Ranganath, S.: ‘Sign language phoneme transcription with rule-based hand trajectory segmentation’, J. Signal Process. Syst., 2010, 59, (2), pp. 211–222 (doi: 10.1007/s11265-008-0292-5).
-
9)
-
31. Guerrero-Curieses, A., Rojo-Álvarez, J.L., Conde-Pardo, P., Landesa-Vazquez, I., Ramos-Lopez, J., Alba-Castro, J.L.: ‘On the performance of kernel methods for skin color segmentation’, EURASIP J. Adv. Signal Process., 2009, 2009, pp. 1–13 (doi: 10.1155/2009/856039).
-
10)
-
21. Yang, R., Sarkar, S., Loeding, B.: ‘Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming’, IEEE Trans. Pattern Anal. Mach. Intell., 2009, 32, pp. 462–477 (doi: 10.1109/TPAMI.2009.26).
-
11)
-
10. Viola, P., Jones, M.: ‘Robust real-time face detection’, Int. J. Comput. Vis., 2004, 2, (57), pp. 137–154 (doi: 10.1023/B:VISI.0000013087.49260.fb).
-
12)
-
19. San-Segundo, R., Barra, R., Cordoba, R., et al: ‘Speech to sign language translation system for Spanish’, Speech Commun., 2008, 50, (11–12), pp. 1009–1020 (doi: 10.1016/j.specom.2008.02.001).
-
13)
-
32. Han, J., Awad, G., Sutherland, A.: ‘Modelling and segmenting subunits for sign language recognition based on hand motion analysis’, Pattern Recognit. Lett., 2009, 30, (6), pp. 623–633 (doi: 10.1016/j.patrec.2008.12.010).
-
14)
-
35. Li, H., Greenspan, M.: ‘Model-based segmentation and recognition of dynamic gestures in continuous video streams’, Pattern Recognit., 2011, 44, (8), pp. 1614–1628 (doi: 10.1016/j.patcog.2010.12.014).
-
15)
-
16)
-
28. Li, H., Greenspan, M.: ‘Continuous time-varying gesture segmentation by dynamic time warping of compound gesture models’. Int. Workshop on Human Activity Recognition and Modelling (HARAM2005), 2005, p. 8.
-
17)
-
5. Grzeszczuk, R., Bradski, G., Chu, M.H., Bouguet, J.: ‘Stereo based gesture recognition invariant to 3D pose and lighting’. IEEE Conf. Computer Vision and Pattern Recognition, , Head Island, SC, USA, 2000, vol. 1, pp. 826–833.
-
18)
-
36. Khan, S., Bailey, D.G., Sen Gupta, G.: ‘Delayed absolute difference (DAD) signatures of dynamic features for sign language segmentation’. Fifth Int. Conf. Automation, Robotics and Applications (ICARA2011), Wellington, New Zealand, 2011, pp. 109–114.
-
19)
-
22. Viblis, M.K., Kyriakopoulos, K.J.: ‘Gesture recognition: the gesture segmentation problem’, J. Intell. Robot. Syst., 2000, 28, pp. 151–158 (doi: 10.1023/A:1008101200733).
-
20)
-
13. Bradski, G.R.: ‘Computer vision face tracking for use in a perceptual user interface’, Intel Technol. J., 1998, 2, (3), pp. 1–15.
-
21)
-
33. Liang, R.-H., Ming, O.: ‘A real-time continuous gesture recognition system for sign language’. IEEE Int. Conf. Automatic Face and Gesture Recognition, Japan, 1998, pp. 558–567.
-
22)
-
26. Kong, W.W., Ranganath, S.: ‘Sign language phoneme transcription with rule-based hand trajectory segmentation’, J. Signal Process. Syst., 2010, 59, (2), pp. 211–222 (doi: 10.1007/s11265-008-0292-5).
-
23)
-
8. Chang, F., Chen, C.-J., Lu, C.-J.: ‘A linear-time component-labeling algorithm using contour tracing technique’, Comput. Vis. Image Underst., 2004, 93, (2), pp. 206–220 (doi: 10.1016/j.cviu.2003.09.002).
-
24)
-
7. Bilal, S., Akmeliawati, R., Momoh, J., Shafie, A.A.: ‘Dynamic approach for real-time skin detection’, J. Real-Time Image Process., 2012, p. 6.
-
25)
-
24. Ong, S.C.W., Ranganath, S.: ‘A new probabilistic model for recognizing signs with systematic modulations’, Third International Workshop on Analysis and Modelling of Faces and Gestures, Rio de Janeiro, Brazil, 2007 (, 4778/2007), pp. 16–30.
-
26)
-
17. Segouat, J.: ‘A study of sign language coarticulation’, Spec. Interest Group Accessible Comput. (SIGACCESS), 2009, 2009, (93), pp. 31–38 (doi: 10.1145/1531930.1531935).
-
27)
-
28)
-
34. Li, H., Greenspan, M.: ‘Multi-scale gesture recognition from time-varying contours’. 10th IEEE Int. Conf. Computer Vision, ICCV 2005, 2005, vol. 1, pp. 236–243.
-
29)
-
25. Ruiduo, Y., Sarkar, S.: ‘Detecting coarticulation in sign language using conditional random fields’. 18th Int. Conf. Pattern Recognition, 2006, ICPR 2006, 2006, vol. 2, pp. 108–112.
-
30)
-
4. Yang, M.H., Ahuja, N.: ‘Recognizing hand gesture using motion trajectories’. IEEE Conf. Computer Vision and Pattern Recognition, , Fort Collins, CO, USA, 1999, vol. 1, pp. 466–483.
-
31)
-
32)
-
21. Yang, R., Sarkar, S., Loeding, B.: ‘Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming’, IEEE Trans. Pattern Anal. Mach. Intell., 2009, 32, pp. 462–477 (doi: 10.1109/TPAMI.2009.26).
-
33)
-
2. MacKenzie, I.S.: ‘Input devices and interaction techniques for advanced computing, in virtual environments and advanced interface design’ (Oxford University Press, Oxford, UK, 1995), pp. 437–470.
-
34)
-
29. Starner, T., Pentland, A.: ‘Real time American sign language recognition from video using hidden Markov model’. Int. Symp. Computer Vision, Florida, USA, 1995, pp. 265–270.
-
35)
-
23. Kahol, K., Tripathi, P., Panchanathan, S., Rikakis, T.: ‘Gesture segmentation in complex motion sequences’. IEEE Int. Conf. Automatic Face and Gesture Recognition, Seoul, Korea, 2004, vol. 3, pp. II-105–8.
-
36)
-
16. Segouat, J., Braffort, A.: ‘Toward modeling sign language coarticulation’, in Kopp, S., Wachsmuth, I. (Eds.): ‘Gesture in embodied communication and human–computer interaction’ (Springer Berlin Heidelberg, 2010), pp. 325–336.
-
37)
-
P. Viola ,
M. Jones
.
Robust real-time face detection.
Int. J. Comput. Vis.
,
2 ,
137 -
154
-
38)
-
J. Han ,
G. Awad ,
A. Sutherland
.
Modelling and segmenting subunits for sign language recognition based on hand motion analysis.
Pattern Recognit. Lett.
,
623 -
633
-
39)
-
20. Alon, J., Athitsos, V., Quan, Y., Sclaroff, S.: ‘A unified framework for gesture recognition and spatiotemporal gesture segmentation’, IEEE Trans. Pattern Anal. Mach. Intell., 2009, 21, pp. 1685–1699 (doi: 10.1109/TPAMI.2008.203).
-
40)
-
9. Imagawa, K., Lu, S., Igi, S.: ‘Color-based hands tracking system for sign language recognition’. Third IEEE Int. Conf. Automatic Face and Gesture Recognition, Nara, Japan, 1998, pp. 462–467.
-
41)
-
42)
-
31. Guerrero-Curieses, A., Rojo-Álvarez, J.L., Conde-Pardo, P., Landesa-Vazquez, I., Ramos-Lopez, J., Alba-Castro, J.L.: ‘On the performance of kernel methods for skin color segmentation’, EURASIP J. Adv. Signal Process., 2009, 2009, pp. 1–13 (doi: 10.1155/2009/856039).
-
43)
-
18. San-Segundo, R., Pardo, J.M., Ferreiros, J., et al: ‘Spoken Spanish generation from sign language’, Interact. Comput., 2010, 22, (2), pp. 123–139 (doi: 10.1016/j.intcom.2009.11.011).
-
44)
-
3. Pavlovic, V.I., Sharma, R., Huang, T.S.: ‘Visual interpretation of hand gestures for human–computer interaction: a review’, IEEE Trans. Pattern Anal. Mach. Intell., 1997, 19, (7), pp. 677–695 (doi: 10.1109/34.598226).
-
45)
-
15. Starner, T.E., Pentland, A.: ‘Real-time American sign language recognition from video using hidden Markov models’. IEEE Int. Symp. Computer Vision, Coral Gables, FL, USA, 1995, pp. 265–270.
-
46)
-
27. Li, H., Greenspan, M.: ‘Segmentation and recognition of continuous gestures’. IEEE Int. Conf. Image Processing, 2007, ICIP 2007, 2007, vol. 1, pp. 365–368.
-
47)
-
30. Vogler, C.P.: ‘American sign language recognition: reducing the complexity of the task with phoneme-based modeling and parallel hidden Markov models’. PhD dissertation, University of Pennsylvania, USA, p. 172.
-
48)
-
14. Bilal, S., Akmeliawati, R., Shafie, A.A., Salami, M.J.E.: ‘Modelling of human upper body for sign language recognition’. Fifth Int. Conf. Automation, Robotics and Applications (ICARA), Wellington, New Zealand, 2011, pp. 104–108.
-
49)
-
19. San-Segundo, R., Barra, R., Cordoba, R., et al: ‘Speech to sign language translation system for Spanish’, Speech Commun., 2008, 50, (11–12), pp. 1009–1020 (doi: 10.1016/j.specom.2008.02.001).
-
50)
-
10. Bilal, S., Akmeliawati, R., Salami, M.J.E., Shafie, A.A., Bouhabba, E.M., et al: ‘A hybrid method using Haar-like and skin-color algorithm for hand posture detection, recognition and tracking’. Int. Conf. Mechatronics and Automation (ICMA), Xi'an, China, 2010, pp. 934–939.
http://iet.metastore.ingenta.com/content/journals/10.1049/joe.2014.0039
Related content
content/journals/10.1049/joe.2014.0039
pub_keyword,iet_inspecKeyword,pub_concept
6
6