access icon openaccess Gesture recognition for transhumeral prosthesis control using EMG and NIR

This is an open access article published by the IET and Zhejiang University Press under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/)
Received 27/03/2020, Accepted 22/06/2020, Revised 09/06/2020, Published 22/06/2020

A key challenge associated with myoelectric prosthesis limbs is the acquisition of a good quality gesture intent signal from the residual anatomy of an amputee. In this study, the authors aim to overcome this limitation by observing the classification accuracy of the fusion of wearable electromyography (EMG) and near-infrared (NIR) to classify eight hand gesture motions across 12 able-bodied participants. As part of the study, they investigate the classification accuracy across a multi-layer perceptron neural network, linear discriminant analysis and quadratic discriminant analysis for different sensing configurations, i.e. EMG-only, NIR-only and EMG-NIR. A separate offline ultrasound scan was conducted as part of the study and served as a ground truth and contrastive basis for the results picked up from the wearable sensors, and allowed for a closer study of the anatomy along the humerus during gesture motion. Results and findings from the work suggest that it could be possible to further develop transhumeral prosthesis using affordable, ergonomic and wearable EMG and NIR sensing, without the need for invasive neuromuscular sensors or further hardware complexity.

Inspec keywords: linear discriminant analysis; multilayer perceptrons; signal classification; medical signal processing; gesture recognition; medical control systems; artificial limbs; electromyography

Other keywords: myoelectric prosthesis limbs; contrastive basis; amputee; gesture recognition; multilayer perceptron neural network; ergonomic EMG; wearable sensors; trans-humeral prosthesis control; able-bodied participants; affordable EMG; good quality gesture intent signal; wearable EMG; linear discriminant analysis; quadratic discriminant analysis; classification accuracy; wearable electromyography; offline ultrasound scan; sensing configurations; ground truth; residual anatomy; hand gesture motions; NIR sensing; EMG-NIR

Subjects: Optical and laser radiation (biomedical imaging/measurement); Probability theory, stochastic processes, and statistics; Biology and medical computing; Digital signal processing; Signal processing and detection; Electrodiagnostics and other electrical measurement techniques; Prosthetics and orthotics; Bioelectric signals; Other topics in statistics; Neural computing techniques; Prosthetics and other practical applications; Prosthetic and orthotic control systems; Optical and laser radiation (medical uses); Other topics in statistics; Electrical activity in neurophysiological processes; Patient diagnostic methods and instrumentation

References

    1. 1)
      • 38. Ju, Z., Ouyang, G., Wilamowska-Korsak, M., et al: ‘Surface EMG based hand manipulation identification via nonlinear feature extraction and classification’, IEEE Sens. J., 2013, 13, (9), pp. 3302–3311.
    2. 2)
      • 19. Niedermeyer, E., da Silva, F.L.: ‘Electroencephalography: basic principles, clinical applications, and related fields’ (Lippincott Williams & Wilkins, Philadelphia, USA, 2005, 5th edn.).
    3. 3)
      • 51. Fougner, A., Stavdahl, O., Kyberd, P. J., et al: ‘Control of upper limb prostheses: terminology and proportional myoelectric control—A review’, IEEE Trans. Neural Syst. Rehabil. Eng., 2012, 20, (5), pp. 663–677.
    4. 4)
      • 10. Guo, W., Yao, P., Sheng, X., et al: ‘An enhanced human-computer interface based on simultaneous sEMG and NIRS for prostheses control’. IEEE Int. Conf. Inf. Autom. (ICIA), Hailar, China, July 2014, pp. 204–207.
    5. 5)
      • 62. Englehart, K., Hudgins, B.: ‘A robust, real-time control scheme for multifunction myoelectric control’, IEEE Trans. Biomed. Eng., 2003, 50, (7), pp. 848–854.
    6. 6)
      • 22. Schlogl, A., Slater, M., Pfurtscheller, G.: ‘Presence research and EEG’. Proc. 5th Annu. Int. Workshop PRESENCE, Porto, Portugal, October 2002.
    7. 7)
      • 4. Kamen, G.: ‘Electromyographic kinesiology’, in ‘Research methods in biomechanics’ (Human Kinetics, Champaign, IL, 2004) eBook: https://www.worldcat.org/title/research-methods-in-biomechanics/oclc/644316055, pp. 179–193.
    8. 8)
      • 50. Bunce, S. C., Izzetoglu, M., Izzetoglu, K., et al: ‘Functional near-infrared spectroscopy’, IEEE Eng. Med. Biol. Mag., 2006, 25, (4), pp. 54–62.
    9. 9)
      • 75. ‘Data Science Blog’, [Online]. Available at https://www.datascienceblog.net/post/machine-learning/linear-discriminant-analysis/, Accessed 8 December 2019.
    10. 10)
      • 40. Hogrel, J.-Y.: ‘Clinical applications of surface electromyography in neuromuscular disorders’, Neurophysiol. Clin., 2005, 35, (2–3), pp. 59–71.
    11. 11)
      • 43. Jiang, N., Dosen, S., Müller, K.-R., et al: ‘Myoelectric control of artificial limbs—Is there a need to change focus?’, IEEE Signal Process. Mag., 2012, 29, (5), pp. 150–152.
    12. 12)
      • 58. ‘Mac sources’ https://macsources.com/myo-armband-available-on-amazon/, accessed 14 August 2019.
    13. 13)
      • 21. Kübler, A., Furdea, A., Halder, S., et al: ‘A brain-computer interface controlled auditory event-related potential (P300) spelling system for locked-in patients’, Annl. N.Y. Acad. Sci., 2009, 1157, pp. 90–100.
    14. 14)
      • 45. Herrmann, S., Attenberger, A., Buchenrieder, K.: ‘Prostheses control with combined near-infrared and myoelectric signals’. Computer Aided Systems Theory - EUROCAST 2011, Part II, ser.LNCS, 6928), 2011, pp. 602–609.
    15. 15)
      • 49. Izzetoglu, M., Izzetoglu, K., Bunce, S., et al: ‘Functional near-infrared neuroimaging’, IEEE Trans. Neural Syst. Rehabil. Eng., 2005, 13, (2), pp. 153–159.
    16. 16)
      • 3. Fumero, R., Costantino, M.L.: ‘Organi artificiali’, in ‘Storia della bioingegneria’ (Patron, Bologna, 2001), pp. 341–365.
    17. 17)
      • 59. Baker, W.B., Parthasarathy, A.B., Busch, D.R., et al: ‘Modified beer-Lambert law for blood flow’, Biomed. Opt. Express, 2014, 5, (11), pp. 4053–4075.
    18. 18)
      • 31. Guo, W., Sheng, X., Liu, H., et al: ‘Development of a multi-channel compact-size wireless hybrid sEMG/NIRS sensor system for prosthetic manipulation’, IEEE Sens. J., 2016, 16, pp. 447–456.
    19. 19)
      • 25. Ohnishi, K., Weir, R.F., Kuiken, T.A.: ‘Neural machine interfaces for controlling multifunctional powered upper-limb prostheses’, Expert Rev. Med. Devices, 2007, 4, (1), pp. 43–53.
    20. 20)
      • 42. Farina, D., Jiang, N., Rehbaum, H., et al: ‘The extraction of neural information from the surface EMG for the control of upper-limb prostheses: emerging avenues and challenges’, IEEE Trans. Neural Syst. Rehabil. Eng., 2014, 22, (4), pp. 797–809.
    21. 21)
      • 46. Chance, B., Anday, E., Nioka, S., et al: ‘A novel method for fast imaging of brain function, non-invasively, with light’, Opt. Express, 1998, 2, pp. 411–423.
    22. 22)
      • 7. Cooperberg, P.L., Tsang, I., Truelove, L., et al: ‘Gray scale ultrasound in the evaluation of rheumatoid arthritis of the knee’, Radiology, 1978, 126, (3), pp. 759–763.
    23. 23)
      • 11. Boushel, R., Langberg, H., Olesen, J., et al: ‘Monitoring tissue oxygen availability with near infrared spectroscopy (NIRS) in health and disease’, Scand. J. Med. Sci. Sports, 2001, 11, (4), pp. 213–222.
    24. 24)
      • 5. Cram, J.R., Kasman, G.S., Holtz, J.: ‘Introduction to surface electromyography’ (Aspen Publishers, Gaithersburg, Md, 1998).
    25. 25)
      • 53. Gaudet, G., Raison, M., Achiche, S., et al: ‘Classification of upper limb phantom movements in transhumeral amputees using electromyographic and kinematic features’, Eng. Appl. Artif. Int., 2018, 68, pp. 153–164.
    26. 26)
      • 56. Clancy, E.A., Morin, E.L., Merletti, R.: ‘Sampling, noise-reduction and amplitude estimation issues in surface electromyography’, J. Electromyogr. Kinesiol., 2002, 12, pp. 1–16.
    27. 27)
      • 15. Singh, S. P.: ‘Magnetoencephalography: basic principles’, Annl. Indian Acad. Neurol., 2014, 17, (Suppl. 1), pp. S107–S112.
    28. 28)
      • 61. ‘3D Hubs’, https://www.3dhubs.com/knowledge-base/pla-vs-abs-whats-difference#what-are-abs-and-pla, accessed 02 June 2019.
    29. 29)
      • 17. Kauhanen, L., Nykopp, T., Sams, M.: ‘Classification of single MEG trials related to left and right index finger movements’, Clin. Neurophysiol., 2006, 117, pp. 430–439.
    30. 30)
      • 35. Li, Y., Chen, X., Tian, J., et al: ‘Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors’. Proc. ICMI-MLMI'10, Beijing, China, 2010.
    31. 31)
      • 12. Ferrari, M., Mottola, L., Quaresima, V.: ‘Principles, techniques, and limitations of near infrared spectroscopy’, Can. J. Appl. Physiol., 2004, 29, (4), pp. 463–487.
    32. 32)
      • 65. Hudgins, B., Parker, P., Scott, R. N.: ‘A new strategy for multifunction myoelectric control’, IEEE Trans. Biomed. Eng., 1993, 40, (1), pp. 82–94.
    33. 33)
      • 64. Chang, C.-C., Lin, C.-J.: ‘LIBSVM: A library for support vector machines’, ACM Trans. Intell. Syst. Technol., 2011, 2, (3), Art. No. 27.
    34. 34)
      • 23. Bi, W.L., Olubiyi, O., Golby, A.J.: ‘Neurosurgical treatment planning’, in Newton, H.B. (Ed.): ‘Handbook of neuro-oncology neuroimaging’ (Elsevier, 2016, 2nd edn.), pp. 217–229.
    35. 35)
      • 77. ‘Science Daily’, https://www.sciencedaily.com/terms/adipose_tissue.htm, accessed 21 November 2019.
    36. 36)
      • 18. Bradberry, T.J., Rong, F., Contreras-Vidal, J.L.: ‘Decoding center-out hand velocity from MEG signals during visuomotor adaptation’, Neuroimage, 2009, 47, pp. 1691–1700.
    37. 37)
      • 9. Barry, D.T., Leonard, J.A.Jr, Gitter, A.J., et al: ‘Acoustic myography as a control signal for an externally powered prosthesis’, Arch. Phys. Med. Rehabil., 1986, 67, pp. 267–269.
    38. 38)
      • 39. Ouyang, G., Zhu, X., Ju, Z., et al: ‘Dynamical characteristics of surface EMG signals of hand grasps via recurrence plot’, IEEE J. Biomed. Health Inf., 2014, 18, (1), pp. 257–265.
    39. 39)
      • 33. Guo, W. C., Sheng, X. J., Liu, H. G.: ‘Toward an enhanced human-machine interface for upper-limb prosthesis control with combined EMG and NIRS signals’, IEEE Trans. Human-Mach. Syst., 2017, 47, (4), pp. 564–575.
    40. 40)
      • 55. Oskeoei, M. A., Hu, H.: ‘Myoelectric control systems—a survey’, Biomed. Signal Proc. Control, 2007, 2, pp. 275–294.
    41. 41)
      • 8. Guo, W., Sheng, X., Liu, H., et al: ‘Mechanomyography assisted myoeletric sensing for upper-extremity prostheses: A hybrid approach’, IEEE Sensors J., 2017, 17, (10), pp. 3100–3108.
    42. 42)
      • 63. Huang, H., Kuiken, T. A., Lipschutz, R. D.: ‘A strategy for identifying locomotion modes using surface electromyography’, IEEE Trans. Biomed. Eng., 2009, 56, (1), pp. 65–73.
    43. 43)
    44. 44)
      • 37. Reiter, R.: ‘Eine neue elektrokunsthand’, Grenzgeb. Med., 1948, 1, (4), pp. 133–135.
    45. 45)
      • 30. Tombini, M., Rigosa, J., Zappasodi, F., et al: ‘Combined analysis of cortical (EEG) and nerve stump signals improves robotic hand control’, Neurorehabil. Neural Repair, 2012, 26, (3), pp. 275–281.
    46. 46)
      • 54. Reaz, M.B.I., Hussain, M. S., Mohd-Yasin, F.: ‘Techniques of EMG signal analysis: detection, processing, classification and applications’, Biol. Proced. Online, 2006, 8, pp. 11–35.
    47. 47)
      • 76. ‘Penn State University’, [Online]. https://onlinecourses.science.psu.edu/stat508/book/export/html/696, Accessed 03 December 2019.
    48. 48)
      • 67. Jarrassé, N., Nicol, C., Touillet, A., et al: ‘Classification of phantom finger, hand, wrist, and elbow voluntary gestures in transhumeral amputees with sEMG’, IEEE Trans. Neural Syst. Rehabil. Eng., 2017, 25, (1), pp. 68–77.
    49. 49)
      • 74. Cybenko, G.: ‘Approximation by superposition of a sigmoidal function’, Math. Control Signal Syst., 1989, 2, pp. 303–314.
    50. 50)
      • 66. Pulliam, C.L., Lambrecht, J.M., Kirsch, R.F.: ‘Electromyogram-based neural network control of transhumeral prostheses’, J. Rehabil. Res. Dev., 2011, 48, (6), pp. 739–754.
    51. 51)
      • 28. Woodward, R., Shefelbine, S., Vaidyanathan, R.: ‘Integrated grip switching and grasp control for prosthetic hands using fused inertial and mechanomyography measurement’. Swarm/Human Blended Intelligence Workshop (SHBI), Cleveland, OH, USA, 2015, pp. 1–8.
    52. 52)
      • 34. Zhang, X., Chen, X., Hui, W. W., et al: ‘Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors’. Proc. 13th Int. Conf. Intell. User Interfaces (IUI), Sanibel Island, USA, 2009, pp. 401–406.
    53. 53)
      • 2. ‘Limbless Statistics’, http://www.limbless-statistics.org/, accessed 10 July 2019.
    54. 54)
      • 29. Rossini, L., Rossini, P. M.: ‘Combining ENG and EEG integrated analysis for better sensitivity and specificity of neuroprosthesis operations’. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Buenos Aires, Argentina, 2010, pp. 134–137.
    55. 55)
      • 70. Montesinos, L., Castaldo, R., Pecchia, L.: ‘On the use of approximate entropy and sample entropy with centre of pressure time-series’, J. NeuroEng. Rehabil., 2018, 15, (116), doi: https://doi.org/10.1186/s12984-018-0465-9.
    56. 56)
      • 72. Kang, W., Shiu, J., Cheng, C. K., et al: ‘Real time implementation of cepstrum coefficient and maximum likelihood method in EMG pattern recognition’, IEEE Trans. Biomed. Eng., 1995, 42, pp. 777–785.
    57. 57)
      • 14. McIntosh, J., McNeill, C., Fraser, M., et al: ‘EMPress: practical hand gesture classification with wrist-mounted EMG and pressure sensing’. Proc. SIGCHI Conf. Human Factors in Computing Systems, San José, USA, May, 2016, pp. 2322–2342.
    58. 58)
      • 71. Boostani, R., Moradi, M.: ‘Evaluation of the forearm EMG signal features for the control of a prosthetic hand’, Physiol. Meas., 2003, 24, (2), pp. 309–319.
    59. 59)
      • 27. Scheme, E., Fougner, A., Stavdahl, Ø., et al: ‘Examining the adverse effects of limb position on pattern recognition based myoelectric control’. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Buenos Aires, Argentina, 2010.
    60. 60)
      • 36. Ouamer, M., Boiteux, M., Petitjean, M., et al: ‘Acoustic myography during voluntary isometric contraction reveals non-propagative lateral vibration’, J. Biomech., 1999, 32, pp. 1279–1285.
    61. 61)
    62. 62)
      • 32. Attenberger, A., Wojciechowski, S.: ‘A real-time classification system for upper limb prosthesis control in MATLAB’, Computer Aided Systems Theory – EUROCAST 2017, Springer, 2018, (LNCS, 10672), pp. 359–365.
    63. 63)
      • 16. ‘University of Washington’, http://ilabs.washington.edu/what-magnetoencephalography-meg, accessed 17 October 2019.
    64. 64)
      • 13. Dementyev, A., Paradiso, J. A.: ‘Wristflex: low-power gesture input with wrist-worn pressure sensors’. Proc. 27th Annual ACM Symp. on User Interface Software and Technology (UIST), Honolulu, USA, October 2014, pp. 161–166.
    65. 65)
      • 20. Schenk, J.F.: ‘The role of magnetic susceptibility in magnetic resonance imaging: MRI magnetic compatibility of the first and second kinds’, Med. Phys., 1996, 23, (6), pp. 815–850.
    66. 66)
      • 69. Graupe, J., Salahi, J., Zhang, D.: ‘Stochastic analysis of electromyogram temporal signatures for multifunctional signal-site activation of prostheses and orthoses’, J. Biomed. Eng., 1985, 7, pp. 18–28.
    67. 67)
      • 52. McIntosh, J., Marzo, A., Fraser, M.: ‘SensIR: detecting hand gestures with a wearable bracelet using infrared transmission and reflection’. Proc. 30th Annu. ACM Symp. on User Interface Software and Technology (UIST), Quebec City, Canada, October 2017, pp. 593–597.
    68. 68)
      • 41. Cifrek, M., Medved, V., Tonkovíc, S., et al: ‘Surface EMG based muscle fatigue evaluation in biomechanics’, Clin. Biomech., 2009, 24, (4), p. pp. 327–340.
    69. 69)
      • 47. Gelmetti, A., Giardini, M.E., Lago, P., et al: ‘Preliminary study of muscle contraction assessment by NIR spectroscopy’, Proc. SPIE, 1998, 3199, pp. 61–67.
    70. 70)
      • 73. ‘Machine Learning Mastery’, https://machinelearningmastery.com/neural-networks-crash-course/, Accessed 20 November 2019.
    71. 71)
      • 6. Fang, Y., Hettiarachchi, N., Zhou, D., et al: ‘Multi-modal sensing techniques for interfacing hand prostheses: a review’, IEEE Sensors J., 2015, 15, (11), pp. 6065–6076.
    72. 72)
      • 24. Fifer, M.S., Acharya, S., Benz, H.L., et al: ‘Toward electrocorticographic control of a dexterous upper limb prosthesis: building brain-machine interfaces’, IEEE. Pulse., 2012, 3, (1), pp. 38–42.
    73. 73)
      • 44. Herrmann, S., Buchenrieder, K.: ‘Fusion of myoelectric and near-infrared signals for prostheses control’. Proc. 4th Int. Convention on Rehabilitation Engineering & Assistive Technology iCREATe'10. Kaki Bukit TechPark II, Singapore Therapeutic, Assistive & Rehabilitative Technologies (START) Centre, Singapore, 2010.
    74. 74)
      • 68. ‘PSGMiner’, http://www.psgminer.com/help/emg_features__.htm?ms=AAA=&mw=MjQw&st=MA==&sct=NzU3, accessed 13 November 2019.
    75. 75)
      • 57. ‘Myo Blog’https://developerblog.myo.com/tag/myo-connect/, accessed 13 August 2019.
    76. 76)
      • 48. Ferrari, M., Quaresima, V.: ‘A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application’, Neuroimage, 2012, 63, (2), pp. 921–935.
    77. 77)
      • 26. Dhillon, G.S., Krüger, T.B., Sandhu, J.S., et al: ‘Effects of short-term training on sensory and motor function in severed nerves of long-term human amputees’, J. Neurophysiol., 2005, 93, pp. 2625–2633.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-csr.2020.0008
Loading

Related content

content/journals/10.1049/iet-csr.2020.0008
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading