Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free Vision-based game design and assessment for physical exercise in a robot-assisted rehabilitation system

© The Institution of Engineering and Technology
Received 16/02/2017, Accepted 13/11/2017, Revised 18/10/2017, Published 14/11/2017

Engagement is a key factor in gaming. Especially, in gamification applications, users’ engagement levels have to be assessed in order to determine the usability of the developed games. The authors first present computer vision-based game design for physical exercise. All games are played with gesture controls. The authors conduct user studies in order to evaluate the perception of the games using a game engagement questionnaire. Participants state that the games are interesting and they want to play them again. Next, as a use case, the authors integrate one of these games into a robot-assisted rehabilitation system. The authors perform additional user studies by employing self-assessment manikin to assess the difficulty levels that can range from boredom to excitement. The authors observe that with the increasing difficulty level, users’ arousal increases. Additionally, the authors perform psychophysiological signal analysis of the participants during the execution of the game under two distinctive difficulty levels. The authors derive features from the signals obtained from blood volume pulse (BVP), skin conductance, and skin temperature sensors. As a result of analysis of variance and sequential forward selection, the authors find that changes in the temperature and frequency content of BVP provide useful information to estimate the players’ engagement.

Inspec keywords: game theory; robot vision

Other keywords: vision based game design; physical exercise; skin conductance; gamification applications; robot assisted rehabilitation system; robot-assisted rehabilitation system; sequential forward selection; game design; skin temperature sensors; self-assessment manikin; BVP; gesture controls; blood volume pulse; computer vision

Subjects: Computer vision and image processing techniques; Robotics; Optical, image and video signal processing; Game theory; Game theory

References

    1. 1)
      • 15. Guadagnoli, M.A., Lee, T.D.: ‘Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning’, J. Motor Behav., 2004, 36, (2), pp. 212–224.
    2. 2)
      • 18. Mihelj, M., Novak, D., Munih, M.: ‘Emotion-aware system for upper extremity rehabilitation’. Virtual Rehabilitation Int. Conf., 2009, 2009, pp. 160–165.
    3. 3)
      • 35. Chaouachi, M., Chalfoun, P., Jraidi, I., et al: ‘Affect and mental engagement: towards adaptability for intelligent systems’. Proc. 23rd Int. Florida Artificial Intelligence Research Society (FLAIRS) Conf., Daytona Beach, FL, 2010.
    4. 4)
      • 36. Niculescu, A., Cao, Y., Nijholt, A.: ‘Stress and cognitive load in multimodal conversational interactions’. Int. Conf. on Human–Computer Interaction, 2009, pp. 891–895.
    5. 5)
      • 8. Lee, S.-H., Oh, S.-H.: ‘A kinect sensor based windows control interface’, Int. J. Control Autom., 2014, 7, (3), pp. 113–124.
    6. 6)
      • 6. Burke, J.W., McNeill, M., Charles, D.K., et al: ‘Optimising engagement for stroke rehabilitation using serious games’, Vis. Comput., 2009, 25, (12), p. 1085.
    7. 7)
      • 37. Günther, A., Witte, O., Hoyer, D.: ‘Autonomic dysfunction and risk stratification assessed from heart rate pattern’, Open Neurol. J., 2010, 4, p. 39.
    8. 8)
      • 39. Gokay, R., Masazade, E., Aydin, C., et al: ‘Emotional state and cognitive load analysis using features from BVP and SC sensors’. IEEE Int. Conf. on Multisensor Fusion and Integration for Intelligent Systems (MFI), 2015, pp. 178–183.
    9. 9)
      • 49. Freeman, J.V., Dewey, F.E., Hadley, D.M., et al: ‘Autonomic nervous system interaction with the cardiovascular system during exercise’, Prog. Cardiovasc. Dis., 2006, 48, (5), pp. 342–362.
    10. 10)
      • 47. Healey, J.A.: ‘Wearable and automotive systems for affect recognition from physiology’. PhD dissertation, Massachusetts Institute of Technology, 2000.
    11. 11)
      • 30. Ozkul, F., Barkana, D.E., Demirbas, S.B., et al: ‘Evaluation of elbow joint proprioception with RehabRoby: a pilot study’, Acta Orthop. Traumatol. Turc., 2012, 46, (5), pp. 332–338.
    12. 12)
      • 19. Badesa, F.J., Morales, R., Garcia-Aracil, N., et al: ‘Multimodal interfaces to improve therapeutic outcomes in robot-assisted rehabilitation’, IEEE Trans. Syst. Man Cybern. C, Appl. Rev., 2012, 42, (6), pp. 1152–1158.
    13. 13)
      • 16. Koenig, A., Omlin, X., Zimmerli, L., et al: ‘Psychological state estimation from physiological recordings during robot-assisted gait rehabilitation’, J. Rehabil. Res. Dev., 2011, 48, (4), pp. 367–386.
    14. 14)
      • 26. ten Holt, G.A., Reinders, M.J., Hendriks, E.: ‘Multi-dimensional dynamic time warping for gesture recognition’. Thirteenth Annual Conf. of the Advanced School for Computing and Imaging, 2007, vol. 300.
    15. 15)
      • 3. Rego, P., Moreira, P.M., Reis, L.P.: ‘Serious games for rehabilitation: a survey and a classification towards a taxonomy’. 5th Iberian Conf. on Information Systems and Technologies, June 2010, pp. 1–6.
    16. 16)
      • 14. Taylor, J.A., Thoroughman, K.A.: ‘Motor adaptation scaled by the difficulty of a secondary cognitive task’, PLoS One, 2008, 3, (6), p. e2485.
    17. 17)
      • 17. Koenig, A., Novak, D., Omlin, X., et al: ‘Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training’, IEEE Trans. Neural Syst. Rehabil. Eng., 2011, 19, (4), pp. 453–464.
    18. 18)
      • 38. Hirshfield, L.M., Bobko, P., Barelka, A., et al: ‘Using noninvasive brain measurement to explore the psychological effects of computer malfunctions on users during human-computer interactions’, Adv. Hum. Comput. Interact., 2014, 2014, p. 2.
    19. 19)
      • 12. Maclean, N., Pound, P.: ‘A critical review of the concept of patient motivation in the literature on physical rehabilitation’, Soc. Sci. Med., 2000, 50, (4), pp. 495–506.
    20. 20)
      • 54. Ekenel, H.K., Sankur, B.: ‘Feature selection in the independent component subspace for face recognition’, Pattern Recognit. Lett., 2004, 25, (12), pp. 1377–1388.
    21. 21)
      • 50. Fung, M.T., Raine, A., Loeber, R., et al: ‘Reduced electrodermal activity in psychopathy-prone adolescents’, J. Abnorm. Psychol., 2005, 114, (2), p. 187.
    22. 22)
      • 25. Wilson, A.D., Bobick, A.F.: ‘Hidden Markov models for modeling and recognizing gesture under variation’, Int. J. Pattern Recognit. Artif. Intell., 2001, 15, (01), pp. 123–160.
    23. 23)
      • 9. Virtualware: ‘Virtual rehabilitation software’, 2014. Available at http://www.virtualrehab.info/.
    24. 24)
      • 40. Dawson, M.E., Schell, A.M., Filion, D.L.: ‘The electrodermal system’, in Cacioppo, T., et al (Ed.): ‘Handbook of psychophysiology’ (Cambridge University Press, 2007, 3rd edn.), pp. 159–181.
    25. 25)
      • 32. Ozkul, F., Barkana, D.E., Maşazade, E.: ‘Admittance filter parameter adjustment of a robot-assisted rehabilitation system (RehabRoby)’, in ‘Mechatronics and robotics engineering for advanced and intelligent manufacturing’ (Springer, 2017), pp. 87–96.
    26. 26)
      • 13. Kaelin-Lang, A., Sawaki, L., Cohen, L.G.: ‘Role of voluntary drive in encoding an elementary motor memory’, J. Neurophysiol., 2005, 93, (2), pp. 1099–1103.
    27. 27)
      • 5. Psaltis, A., Kaza, K., Stefanidis, K., et al: ‘Multimodal affective state recognition in serious games applications’. 2016 IEEE Int. Conf. on Imaging Systems and Techniques (IST), October 2016, pp. 435–439.
    28. 28)
      • 48. Clifford, G.D.: ‘Signal processing methods for heart rate variability’. PhD dissertation, Department of Engineering Science, University of Oxford, 2002.
    29. 29)
      • 45. Ohsuga, M., Shimono, F., Genno, H.: ‘Assessment of phasic work stress using autonomic indices’, Int. J. Psychophysiol., 2001, 40, (3), pp. 211–220.
    30. 30)
      • 52. Kim, K.H., Bang, S., Kim, S.: ‘Emotion recognition system using short-term monitoring of physiological signals’, Med. Biol. Eng. Comput., 2004, 42, (3), pp. 419–427.
    31. 31)
      • 22. Novak, D., Mihelj, M., Ziherl, J., et al: ‘Psychophysiological measurements in a biocooperative feedback loop for upper extremity rehabilitation’, IEEE Trans. Neural Syst. Rehabil. Eng., 2011, 19, (4), pp. 400–410.
    32. 32)
      • 42. Mulder, L.J.M., van Roon, A.M.: ‘A psychophysiological approach to working conditions’, in Backs, R.W., Boucsein, W. (Eds.): ‘Engineering Psychophysiology’ (Lawrence Erlbaum Associates, 2000), pp. 139–160.
    33. 33)
      • 53. Rani, P., Sarkar, N., Smith, C.A., et al: ‘Anxiety detecting robotic system–towards implicit human–robot collaboration’, Robotica, 2004, 22, (01), pp. 85–95.
    34. 34)
      • 27. R. Solutions: ‘Kinect with MS-SDK’. Available at https://www.assetstore.unity3d.com/en/#!/content/7747.
    35. 35)
      • 7. Unity: ‘Game engine, tools and multiplatform’. Available at https://unity3d.com/unity.
    36. 36)
      • 55. Lin, K., Xia, F., Wang, W., et al: ‘System design for big data application in emotion-aware healthcare’, IEEE Access, 2016, 4, pp. 6901–6909.
    37. 37)
      • 1. Wang, C., Liu, Z., Fels, S.: ‘Everyone can do magic: an interactive game with speech and gesture recognition’. Int. Conf. on Entertainment Computing, 2010, pp. 32–42.
    38. 38)
      • 51. Kreibig, S.D.: ‘Autonomic nervous system activity in emotion: a review’, Biol. Psychol., 2010, 84, (3), pp. 394–421.
    39. 39)
      • 41. Andreassi, J.L.: ‘Psychophysiology: human behavior & physiological response’ (Psychology Press, 5th edn, 2006).
    40. 40)
      • 31. Ozkul, F., Barkana, D.E.: ‘Upper-extremity rehabilitation robot RehabRoby: methodology, design, usability and validation’, Int. J. Adv. Robot. Syst., 2013, 10, (12), p. 401.
    41. 41)
      • 43. Rousselle, J.G., Blascovich, J., Kelsey, R.M.: ‘Cardiorespiratory response under combined psychological and exercise stress’, Int. J. Psychophysiol., 1995, 20, (1), pp. 49–58.
    42. 42)
      • 2. Ricciardi, F., Paolis, L.T.D.: ‘A comprehensive review of serious games in health professions’, Int. J. Comput. Games Technol., 2014, 2014, p. 9.
    43. 43)
      • 11. Erol Barkana, D., Özkul, F.: ‘A robot-assisted rehabilitation system–RehabRoby’, in ‘Interdisciplinary Mechatronics’ (John Wiley & Sons, Inc., 2013), pp. 145–162.
    44. 44)
      • 33. Lang, P.J., Greenwald, M.K., Bradley, M.M., et al: ‘Looking at pictures: affective, facial, visceral, and behavioral reactions’, Psychophysiology, 1993, 30, (3), pp. 261–273.
    45. 45)
      • 21. Riener, R., Munih, M.: ‘Guest editorial special section on rehabilitation via bio-cooperative control’, IEEE Trans. Neural Syst. Rehabil. Eng., 2010, 4, (18), pp. 337–338.
    46. 46)
      • 28. Brockmyer, J.H., Fox, C.M., Curtiss, K.A., et al: ‘The development of the game engagement questionnaire: a measure of engagement in video game-playing’, J. Exp. Soc. Psychol., 2009, 45, (4), pp. 624–634.
    47. 47)
      • 4. Rego, P.A., Moreira, P.M., Reis, L.P.: ‘Natural user interfaces in serious games for rehabilitation’. 6th Iberian Conf. on Information Systems and Technologies (CISTI 2011), June 2011, pp. 1–4.
    48. 48)
      • 23. Bradley, M.M., Lang, P.J.: ‘Measuring emotion: the self-assessment manikin and the semantic differential’, J. Behav. Ther. Exp. Psychiatry, 1994, 25, (1), pp. 49–59.
    49. 49)
      • 20. Novak, D., Mihelj, M., Munih, M.: ‘A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing’, Interact. Comput., 2012, 24, (3), pp. 154–172.
    50. 50)
      • 24. Chizari, H.: ‘Combining voice and gesture for human computer interaction’. Master's thesis, Freiburg University, 2013.
    51. 51)
      • 34. Swangnetr, M., Kaber, D.B.: ‘Emotional state classification in patient–robot interaction using wavelet analysis and statistics-based feature selection’, IEEE Trans. Hum. Mach. Syst., 2013, 43, (1), pp. 63–75.
    52. 52)
      • 10. Jintronix: ‘Kinect rehabilitation software’. Available at http://www.jintronix.com/.
    53. 53)
      • 46. Jo, J., Lee, Y., Shin, H.S.: ‘Real-time analysis of heart rate variability for a mobile human emotion recognition system’. Recent Advances in Electrical and Computer Engineering, online access 15 March 2015. Available at http://www.wseas.us/elibrary/conferences/2013/CambridgeUSA/ELIT/ELIT-27.pdf.
    54. 54)
      • 44. Boucsein, W.: ‘Electrodermal activity’ (Springer US, 2012, 2nd edn.).
    55. 55)
      • 29. Herbelin, B., Ciger, J., Brooks, A.L.: ‘Customization of gaming technology and prototyping of rehabilitation applications’. Proc. 7th Int. Conf. on Disability, Virtual Reality and Associated Technologies with ArtAbilitation, 2008, pp. 8–11.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cvi.2017.0122
Loading

Related content

content/journals/10.1049/iet-cvi.2017.0122
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address