Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon openaccess Review: Are we stumbling in our quest to find the best predictor? Over-optimism in sensor-based models for predicting falls in older adults

The field of fall risk testing using wearable sensors is bustling with activity. In this Letter, the authors review publications which incorporated features extracted from sensor signals into statistical models intended to estimate fall risk or predict falls in older people. A review of these studies raises concerns that this body of literature is presenting over-optimistic results in light of small sample sizes, questionable modelling decisions and problematic validation methodologies (e.g. inherent problems with the overly-popular cross-validation technique, lack of external validation). There seem to be substantial issues in the feature selection process, whereby researchers select features before modelling begins based on their relation to the target, and either perform no validation or test the models on the same data used for their training. This, together with potential issues related to the large number of features and their correlations, inevitably leads to models with inflated accuracy that are unlikely to maintain their reported performance during everyday use in relevant populations. Indeed, the availability of rich sensor data and many analytical options provides intellectual and creative freedom for researchers, but should be treated with caution, and such pitfalls must be avoided if we desire to create generalisable prognostic tools of any clinical value.

References

    1. 1)
    2. 2)
    3. 3)
    4. 4)
      • 2. http://www.who.int/topics/ageing/en/, accessed 30/3/2015.
    5. 5)
    6. 6)
    7. 7)
    8. 8)
    9. 9)
      • 44. Liu, Y., Redmond, S.J., Shany, T., et al: ‘Validation of an accelerometer-based fall prediction model’. Proc. of Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, Chicago, Illinois, USA, August 2014, pp. 45314534.
    10. 10)
    11. 11)
      • 25. Lord, S.R., Menz, H.B., Tiedemann, A.: ‘A physiological profile approach to falls risk assessment and prevention’, Phys. Ther., 2003, 83, (3), pp. 237252.
    12. 12)
    13. 13)
    14. 14)
    15. 15)
    16. 16)
    17. 17)
      • 54. Vanwinckelen, G., Blockeel, H.: ‘On estimating model accuracy with repeated cross-validation’. Proc. of BeneLearn: The 21st Belgian-Dutch Conf. on Machine Learning, Ghent, Belgium, May 2012, pp. 3944.
    18. 18)
    19. 19)
    20. 20)
    21. 21)
    22. 22)
    23. 23)
    24. 24)
    25. 25)
    26. 26)
    27. 27)
    28. 28)
    29. 29)
      • 36. Berg, K., Wood-Dauphinee, S., Williams, J.: ‘The balance scale: reliability assessment with elderly residents and patients with an acute stroke’, Scand. J. Rehabil. Med., 1995, 27, pp. 2736.
    30. 30)
    31. 31)
    32. 32)
      • 57. Oliver, D., Healy, F.: ‘Falls risk prediction tools for hospital inpatients: do they work?’, Nurs. Times, 2009, 105, (7), pp. 1821.
    33. 33)
    34. 34)
      • 35. Fuke, S., Suzuki, T., Doi, M.: ‘Estimation of falling risk based on acceleration signals during initial gait’. Proc. of Int. Conf. on Biomedical Engineering, Penang, Malaysia, February 2012, pp. 286291.
    35. 35)
    36. 36)
      • 63. Doheny, E.P., Chie Wei, F., Foran, T., Greene, B.R., Cunningham, C., Kenny, R.A.: ‘An instrumented sit-to-stand test used to examine differences between older fallers and non-fallers’. Proc. of Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, Boston, Massachusetts, USA, August–September 2011, pp. 30633066.
    37. 37)
    38. 38)
    39. 39)
    40. 40)
      • 21. Mahoney, F., Barthel, D.: ‘Functional evaluation: the barthel index’, Md State Med. J., 1965, 14, pp. 6165.
    41. 41)
    42. 42)
    43. 43)
    44. 44)
    45. 45)
    46. 46)
    47. 47)
    48. 48)
    49. 49)
      • 3. Gillespie, L.D., Robertson, M.C., Gillespie, W.J., Sherrington, C., Gates, S., Clemson, L.M., et al: ‘Interventions for preventing falls in older people living in the community’ (The Cochrane Library, 2012).
    50. 50)
    51. 51)
      • 51. Bengio, Y., Grandvalet, Y.: ‘No unbiased estimator of the variance of k-fold cross-validation’, J. Mach. Learn. Res., 2004, 5, pp. 10891105.
    52. 52)
    53. 53)
    54. 54)
    55. 55)
    56. 56)
    57. 57)
    58. 58)
      • 52. Kohavi, R.: ‘A study of cross-validation and bootstrap for accuracy estimation and model selection’. Proc. of Int. Joint Conf. on Articial Intelligence, Montreal, Quebec, Canada, August 1995, pp. 11371145.
    59. 59)
      • 27. Liu, Y., Redmond, S.J., Narayanan, M.R., Lovell, N.H.: ‘Classification between non-multiple fallers and multiple fallers using a triaxial accelerometry-based system’. Proc. of Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, Boston, Massachusetts, USA, August–September 2011, pp. 14991502.
    60. 60)
    61. 61)
    62. 62)
    63. 63)
    64. 64)
http://iet.metastore.ingenta.com/content/journals/10.1049/htl.2015.0019
Loading

Related content

content/journals/10.1049/htl.2015.0019
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address