As smart audio-visual multimedia devices are developed for various applications, there has been a growing interest in effective human–computer interaction (HCI) interfaces for specific environments. There have also been great efforts to implement HCI interfaces into musical instruments, in which it would be possible to take intuitions, comfort and expressiveness into the musical instruments. However, most of the traditional HCI interfaces are not applicable because both hands are likely to be occupied while playing a musical instrument. In this environment, a lip reading method can be used. A lip reading method is a HCI method that analyses lip motion to recognise spoken words. In this study, a lip reading method is proposed with its application. As a specific example of the interface for musical instruments, the guitar effector application is presented. The proposed lip reading method uses a constrained local model instead of the conventional active appearance model for effective facial feature tracking. The proposed method also uses a dynamic time warping-based classifier for word recognition which is effective for simple real-time implementation of lip reading. The proposed lip reading method shows 85.0% word recognition accuracy on OuluVS database and is effectively applied to the proposed guitar effector application.

References

1. 1)
  - 4. Lyons, M., Haehnel, M., Tetsutani, N.: ‘Designing, playing, and performing with a vision-based mouth interface’. Proc. 2003 Conf. on New Interfaces for Musical Expression, Montreal, Canada, May 2003, pp. 116–121.
2. 2)
  - 17. Chen, Q., Yao, J., Cham, W.: ‘3D model-based pose invariant face recognition from multiple views’, IET Comput. Vis., 2007, 1, (1), pp. 25–34 (doi: 10.1049/iet-cvi:20060014).
3. 3)
  - 8. Bakry, A., Elgammal, A.: ‘MKPLS: manifold kernel partial least squares for lipreading and speaker identification’. Proc. IEEE Computer Vision and Pattern Recognition, Portland, OR, USA, June 2013, pp. 684–691.
4. 4)
  - 10. Viola, P., Jones, M.: ‘Robust real-time face detection’, Int. J. Comput. Vis., 2004, 2, (57), pp. 137–154 (doi: 10.1023/B:VISI.0000013087.49260.fb).
5. 5)
  - 19. Teijeiro-Mosquera, L., Alba-Castro, J.: ‘Performance of active appearance model-based pose-robust face recognition’, IET Comput. Vis., 2011, 5, (6), pp. 348–357 (doi: 10.1049/iet-cvi.2010.0184).
6. 6)
  - 10. Cootes, T., Taylor, C., Cooper, D., Graham, J.: ‘Active shape models – their training and application’, Comput. Vis. Image Underst., 1995, 61, (1), pp. 38–59 (doi: 10.1006/cviu.1995.1004).
7. 7)
  - 1. Cook, P.: ‘Principles for designing computer music controllers’. Proc. 2001 Conf. on New Interfaces for Musical Expression, Seattle, WA, USA, April 2001, pp. 1–4.
8. 8)
  - 9. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: ‘Incremental face alignment in the wild’. Proc. IEEE Computer Vision and Pattern Recognition, Columbus, OH, USA, June 2014, pp. 1859–1866.
9. 9)
  - 16. Jeni, L., Lorincz, A., Nagy, T., et al: ‘3D shape estimation in video sequences provides high precision evaluation of facial expressions’, Image Vis. Comput., 2012, 30, (10), pp. 785–795 (doi: 10.1016/j.imavis.2012.02.003).
10. 10)
  - 12. Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.: ‘Interactive facial feature localization’. Proc. European Conf. on Computer Vision, Florence, Italy, October 2012, pp. 679–692.
11. 11)
  - 5. Shin, J., Lee, J., Kim, D.: ‘Real-time lip reading system for isolated Korean word recognition’, Pattern Recognit., 2011, 44, (3), pp. 559–571 (doi: 10.1016/j.patcog.2010.09.011).
12. 12)
  - 7. Zhao, G., Barnard, M., Pietikainen, M.: ‘Lipreading with local spatiotemporal descriptors’, IEEE Trans. Multimed., 2009, 11, (7), pp. 1254–1265 (doi: 10.1109/TMM.2009.2030637).
13. 13)
  - 13. Wachter, M., Matton, M., Demuynck, K., Wambacq, P., Cools, R., Compernolle, D.: ‘Template-based continuous speech recognition’, IEEE Trans. Audio, Speech, Lang. Process., 2007, 15, (4), pp. 1377–1390 (doi: 10.1109/TASL.2007.894524).
14. 14)
  - 26. Rabiner, L.: ‘A tutorial on hidden Markov models and selected applications in speech recognition’, Proc. IEEE, 1989, 77, (2), pp. 257–286 (doi: 10.1109/5.18626).
15. 15)
  - 2. Tarabella, L., Bertini, G.: ‘Giving expression to multimedia performance’. Proc. ACM Workshops on Multimedia, Los Angeles, CA, USA, October 2000, pp. 35–38.
16. 16)
  - 18. Lan, Y., Theobald, B.-J., Harvey, R.: ‘View independent computer lip-reading’. Proc. IEEE Int. Conf. on Multimedia and Expo, Melbourne, Australia, July 2012, pp. 432–437.
17. 17)
  - 21. D'Ausilio, A.: ‘Arduino: a low-cost multipurpose lab equipment’, Behavior Res. Methods, 2012, 44, (2), pp. 305–313 (doi: 10.3758/s13428-011-0163-z).
18. 18)
  - 11. Cootes, T., Edwards, G., Taylor, C.: ‘Active appearance models’, IEEE Trans. Pattern Anal. Mach. Intell., 2001, 23, (6), pp. 681–685 (doi: 10.1109/34.927467).
19. 19)
  - 3. Funk, M., Kuwabara, K., Lyons, M.: ‘Sonification of facial actions for musical expression’. Proc. 2005 Conf. on New Interfaces for Musical Expression, Vancouver, Canada, May 2005, pp. 127–131.
20. 20)
  - 6. Su, J., Sarivastava, A., Souza, F., Sarkar, S.: ‘Rate-invariant analysis of trajectories on Riemannian manifolds with application in visual speech recognition’. Proc. IEEE Computer Vision and Pattern Recognition, Columbus, OH, USA, June 2014, pp. 620–627.
21. 21)
  - 15. Baltrusaitis, T., Robinson, P., Morency, L.: ‘3D constrained local model for rigid and non-rigid facial tracking’. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Providence, RI, USA, June 2012, pp. 16–21.

New interface for musical instruments using lip reading

References

Related content