Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

Large-scale software engineering questions – expert opinion or empirical evidence?

Large-scale software engineering questions – expert opinion or empirical evidence?

For access to this article, please select a purchase option:

Buy article PDF
£12.50
(plus tax if applicable)
Add to cart
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)
Add to cart

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
IET Software — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

© The Institution of Engineering and Technology
Published

A recent report on the state of the UK information technology (IT) industry based most of its findings and recommendations on expert opinion. It is surprising that the report was unable to incorporate more empirical evidence. This paper aims to assess whether it is necessary to base IT industry and academic policy on expert opinion rather than on empirical evidence. Current evidence related to the rate of project failure is identified and the methods used to accumulate that evidence discussed. This shows that the report failed to identify relevant evidence and most evidence related to project failure is based on convenience samples. The status of empirical research in the computing disciplines is reviewed showing that empirical evidence covers a restricted range of subjects and seldom addresses the ‘Society’ level of analysis. Other more robust designs that would address large-scale IT questions are discussed. We recommend adopting a more systematic approach to accumulating and reporting evidence. In addition, we propose using quasi-experimental designs developed and used in the social sciences to improve the methodology used for undertaking large-scale empirical studies in software engineering.

Inspec keywords: DP industry; software engineering

Other keywords: large-scale software engineering; UK information technology industry; expert opinion; social sciences; empirical evidence

Subjects: Software engineering techniques; Software management

References

    1. 1)
      • D. Phan , D. Vogel , J.F. Nunamaker . The Search for Perfect project management. Computerworld , 95 - 100
    2. 2)
    3. 3)
      • C. Sauer , C. Cuthbertson . (2003) The State of IT Management in the UK.
    4. 4)
      • Segal, J., Grinyer, A., Sharp, H.: `The type of evidence produced by empirical software engineers', Proceedings of REBSE'05, 2005, ICSE.
    5. 5)
    6. 6)
      • R.L. Glass , I. Vessey , V. Ramesh . Research in software engineering: an analysis of the literature. Inf. Softw. Techol. , 8 , 491 - 506
    7. 7)
      • Jørgensen, M., Dybå, T., Kitchenham, B.: `Teaching Evidence-Based Software Engineering to University Students', 11thIEEE Int. Software Metrics Symp. (METRICS'05), 2005, p. 24.
    8. 8)
      • D.I.K. Sjøberg , J.E. Hannay , O. Hansen , V.B. Kampenes , A. Karahasanovic , N.K. Liborg , A.C. Rekdal . A survey of controlled experiments in software engineering. IEEE Trans. SE , 9 , 733 - 753
    9. 9)
      • F.J. Fowler . (2002) Survey Research Methods.
    10. 10)
      • M.V. Zelkowitz , V. Basili . (2007) Techniques for Empirical validation.
    11. 11)
      • A.M. Jenkins , J.D. Naumann , J.C. Wetherbe . Empirical investigation of systems development practice and results. Inf. Manag. , 73 - 82
    12. 12)
      • C. Mair , M. Shepperd . (2005) The consistency of empirical comparisons of regression and analogy-based software project cost prediction.
    13. 13)
      • D. Phan . (1990) Information Systems project management: an Integrated Resource Planning Perspective Model.
    14. 14)
    15. 15)
      • M. Zelkowitz , D. Wallace . Experimental models for validating computer technology. IEEE Comput. , 5 , 23 - 31
    16. 16)
      • B. Kitchenham , S.L. Pfleeger , L.M. Pickard , P. Jones , D. Hoaglin , K. El Emam , J. Rosenberg . Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Trans. Softw. Eng. , 8 , 721 - 734
    17. 17)
      • Jones, T.C.: `Project management tools and software failures and successes', July 1998, , Crosstalk, available at: http://www.stsc.hill.af.mil/crosstalk/1998/07/tools.asp.
    18. 18)
      • `The Challenges of Complex IT Projects', The Report of a working group from the Royal Academy of Engineering and The British Computer Society, April 2004, The Royal Academy of Engineering.
    19. 19)
      • B. Kitchenham , E. Mendes , G.H. Travassos . A systematic review of cross vs. within-company cost estimation studies. IEEE Trans. SE , 5 , 316 - 329
    20. 20)
      • McGarry, F., Burke, S., Decker, B.: `Measuring the impacts individual process maturity attributes have on software Products', Proc. Fifth Int. Software Metrics Symp., IEEE Computer Society, 1998, p. 52–60.
    21. 21)
      • D.J. Morrow . (2008) Personal communication.
    22. 22)
      • Kitchenham, B., Dybå, T., Jørgensen, M.: `Evidence-based Software Engineering', Proc. 26th Int. Conf. on Software Engineering, (ICSE '04), IEEE Computer Society, 2004, Washington, DC, USA, p. 273–281.
    23. 23)
      • F. Bergeron , J.-Y. St-Arnaud . Estimation of information systems Development efforts: a pilot study. Inf. Manag. , 238 - 254
    24. 24)
      • R.E. Stake . (1995) The Art of Case Study Research.
    25. 25)
      • D. Dalcher , A. Genus . Avoiding IS/IT implementation failure. TASM , 4 , 404 - 407
    26. 26)
      • F.J. Heemstra , R. Kusters , M. van Genuchten . (1989) Selections of software cost estimation models.
    27. 27)
      • M. Jørgensen . A review of studies on expert estimation of software development effort. J. Syst. Softw. , 37 - 60
    28. 28)
      • R.K. Yin . (2003) Case Study Research Design and Methods.
    29. 29)
      • A. Taylor . IT projects sink or swim. Brit. Comput. Soc. Rev.
    30. 30)
    31. 31)
      • V. Ramesh , R.L. Glass , I. Vessey . Research in computer science: an empirical study. JSS , 165 - 176
    32. 32)
      • C. Wohlin , H. Petersson , A. Aurum , N. Juristo , A. Moreno . (2003) Combining data from reading experiments in Software Inspections.
    33. 33)
      • M. Jørgensen . Estimation of software development work effort: evidence on expert judgement and formal models. Int. J. Forecast. , 3 , 449 - 462
    34. 34)
      • M. Jørgensen, , M. Shepperd . A systematic review of software development cost estimation studies. IEEE Trans. Softw. Eng. , 1 , 33 - 53
    35. 35)
      • Moløkken-Østvold, K., Jørgensen, M., Tanilkan, S.S., Gallis, H., Lien, A., Hove, S.: `A Survey on Software Estimation in the Norwegian Industry', Proc. 10th Int. Symp. on Software metrics. Metrics 2004, 2004, IEEE Computer Society, p. 208–219.
    36. 36)
    37. 37)
      • E. Mendes . (2005) A systematic review of Web engineering research.
    38. 38)
      • C. Zannier , G. Melnik , F. Maurer . On the Success of Empirical Studies in the Int. Conf. on Software Engineering'. ICSE06 , 341 - 350
    39. 39)
      • M. Petticrew , H. Roberts . (2006) Systematic Reviews in the Social Sciences. A Practical Guide.
    40. 40)
      • Kitchenham, B.: `Procedures for Performing Systematic Reviews', Joint Technical Report, Keele University TR/SE-0401 and NICTA 0400011T.1, July 2004.
    41. 41)
    42. 42)
      • A. Høfer , W.F. Tichy , V. Basili . (2007) Status of Empirical Research in Software Engineering.
    43. 43)
    44. 44)
      • M. Jørgensen , K. Moløkken-Østvold . How large are software cost overruns? A review of the 1994 CHAOS report. Inf. Softw. Technol. , 297 - 301
    45. 45)
      • Vessey, I., Ramesh, V., Glass, R.L.: `A united classification system for research in the computing disciplines', TR 107-1, 2002, available at: www.indiana.edu/~isdept/research/workingpapers.html.
    46. 46)
    47. 47)
      • K. Moløkken , M. Jørgensen . (2004) Project Estimation in the Norwegian Software Industry – a summary.
    48. 48)
      • W.R. Shaddish , T.D. Cook , D.T. Campbell . (2002) Experimental and Quasi-Experimental Designs for Generalized Causal Inference.
    49. 49)
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-sen_20060052
Loading

Related content

content/journals/10.1049/iet-sen_20060052
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address