Liu, J. L. Y. and Wyatt, Jeremy C.. (2011) The case for randomized controlled trials to assess the impact of clinical information systems. Journal of the American Medical Informatics Association, Vol.18 (No.2). pp. 173-180. ISSN 1067-5027

Full text not available from this repository.

Abstract

There is a persistent view of a significant minority in the medical informatics community that the randomized controlled trial (ROT) has a limited role to play in evaluating clinical information systems. A common reason voiced by skeptics is that these systems are fundamentally different from drug interventions, so the RCT is irrelevant. There is an urgent need to promote the use of RCTs, given the shift to evidence-based policy and the need to demonstrate cost-effectiveness of these systems. The authors suggest returning to first principles and argue that what is required is clarity about how to match methods to evaluation questions. The authors address common concerns about RCTs, and the extent to which they are fallacious, and also discuss the challenges of conducting RCTs in informatics and alternative study designs when randomized trials are infeasible. While neither a perfect nor universal evaluation method, RCTs form an important part of an evaluator's toolkit.

Item Type:	Journal Article
Subjects:	R Medicine > R Medicine (General)
Divisions:	Faculty of Science > WMG (Formerly the Warwick Manufacturing Group)
Library of Congress Subject Headings (LCSH):	Medical informatics, Information storage and retrieval systems -- Medicine -- Evaluation, Medical records -- Data processing, Clinical trials
Journal or Publication Title:	Journal of the American Medical Informatics Association
Publisher:	BMJ Publishing Group
ISSN:	1067-5027
Date:	2011
Volume:	Vol.18
Number:	No.2
Page Range:	pp. 173-180
Identification Number:	10.1136/jamia.2010.010306
Status:	Peer Reviewed
Publication Status:	Published
Funder:	Scottish Funding Council (SFC)
References:	1. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197e203. 2. Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical informatics. J Am Med Inform Assoc 1994;1:353e5. 3. Heathfield HA, Buchan IB. Current evaluations of information technology in health care are often inadequate. BMJ 1996;313:1008. 4. Heathfield HA, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998;316:1959e61. 5. Kaplan B. Evaluating informatics applicationsdsome alternative approaches, and call for methodological pluralism. Int J Med Inform 2001;64:39e56. 6. UK Institute of Health Informatics. Evaluation of Electronic Patient and Health Record Projects. Winchester: ERDIP Programme, NHS Information Authority, 2001. 7. Stoop A, Heathfield H, de Mul M, et al. Evaluation of patient care information systems: theory and practice. In: Berg M, ed. Health Information Management: Integrating Information Technology in Health Care Work. London: Routledge, 2004:206e29. 8. Brender J. Handbook of Evaluation Methods for Health Informatics. London: Elsevier Academic Press, 2006. 9. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systematic evaluation. PLoS Med 2009;6:e10000126. 10. Rigby M. Health informatics as a tool to improve quality in non-acute carednew opportunities and a matching need for a new evaluation paradigm. Int J Med Inform 1999;56:141e50. 11. Patel VL, Kaufmann DR, Arocha JF. Emerging paradigms of cognition in medical decision making. J Biomed Inform 2002;35:52e75. 12. Kaplan B, Shaw NT. People, Organizational, and Social Issues. IMIA Yearbook of Medical Informatics. Heidelberg: Schattauer, 2002:91e102. 13. Beale S, Sanderson D, Kruger J; for the York Health Economics Consortium. Evaluation of the Telecare Development Programme: Final Report. Edinburgh: Scottish Government, 2009. 14. Friedman CP, Wyatt JC. Evaluation Methods in Bio-Medical Informatics. 2nd edn. New York: Springer-Verlag, 2006. 15. Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall, 1991. 16. Pope C, Mays N. Qualitative Research in Health Care. 2nd edn. London: BMJ Publishing, 2000. 17. Rothman KJ, Greenland S. Modern Epidemiology. 2nd edn. Philadelphia: Lippincott-Raven, 1998. 18. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 2006;13:16e23. 19. de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005? Int J Med Inform 2008;77:41e9. 20. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc 2009;16:531e8. 21. Liu JLY, Wyatt JC, Deeks JJ, et al. Systematic reviews of clinical decision tools for acute abdominal pain. Health Technol Assess 2006;10:1e186, iiieiv. 22. Copi IM, Cohen C. Introduction to Logic. 9th edn. Englewood Cliffs, NJ: Prentice- Hall, 1994. 23. Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003;69:251e9. 24. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services. BMJ 1995;311:42e5. 25. Drummond MF, Sculpher MJ, Torrance GW, et al. Methods for the Economic Evaluation of Health Care Programmes. 3rd edn. New York: Oxford University Press, 2005. 26. Pocock SJ. Clinical Trials: A Practical Approach. Chichester: John Wiley and Sons, 1983. 27. Schulz KF, Altman DG, Moher D; the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel-group randomized trials. BMJ 2010;340:c332. 28. Moher D, Hopewell S, Schulz KF, et al. The CONSORT Statement 2010 updated guidelines for reporting randomized trials: explanation and elaboration. BMJ 2010;340:c869. 29. Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultations: a multisite study of 2 systems. JAMA 1999;282:1851e6. 30. Macintyre S, Petticrew M. Good intentions and received wisdom are not enough. J Epidemiol Community Health 2000;54:802e3. 31. Wynder EL. Studies in mechanism and prevention: striking a proper balance. Am J Epidemiol 1994;139:547e9. 32. McLean S, Chandler D, Nurmatov U, et al. Telehealthcare for asthma. Cochrane Database Syst Rev 2010;(10):CD007717. 33. Liu JLY, Wyatt JC. The Impact of Telepsychiatry, Teledermatology, Teleent and Teleradiology on the Health Services: A Rapid Review. Portland, OR: Medicaid Evidence-Based Decisions Program, 2008. 34. Clark JS, Mair FS, O’Donnell C, et al. E-health: implementation and evaluation research in Scotlandda scoping exercise. J Telemed Telecare 2008;14:119e21. 35. Abraham EP, Chain E, Fletcher CM. Further observations on penicillin. Lancet 1941;2:177e88. 36. Peto R, Pike MC, Armitage P, et al. Design and analysis of randomized clinical trials requiring prolonged observation of each patient. I. Introduction and design. Br J Cancer 1976;34:585e612. 37. Collins R, Peto R, Gray R, et al. Large-Scale Randomized Evidence: Trials and Overviews. Oxford Textbook of Medicine. Oxford: Oxford University Press, 1998:21e32. 38. Peto R, Baigent C. Trials: the next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317:1170e1. 39. Collins R, MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. Lancet 2001;357:373e80. 40. Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. Am J Med 1982;72:233e40. 41. Tang JL, Leung PC. An efficacy-driven approach to the research and development of traditional Chinese medicine. Hong Kong Med J 2001;7:375e80. 42. Anon. Effectiveness of health checks conducted by nurses in primary care: final results of the OXCHECK study. Imperial Cancer Research Fund OXCHECK Study Group. BMJ 1995;310:1099e104. 43. Proudfoot J, Guest D, Carson J, et al. Effect of cognitive-behavioural training on job-finding among long-term unemployed people. Lancet 1997;350:96e100. 44. Wyatt JC, Paterson-Brown S, Johanson R, et al. Trial of outreach visits to enhance the use of systematic reviews in 25 obstetric units. BMJ 1998;317:1041e6. 45. Hobbs M, Mayou R, Harrison B, et al. A randomised controlled trial of psychological debriefing for victims of road traffic accidents. BMJ 1996;313:1438e9. 46. Allen K, Blascovich J. The value of service dogs for people with severe ambulatory disabilities. A randomised controlled trial. JAMA 1996;275:1001e6. 47. Byrd RC. Positive therapeutic effects of intercessory prayer in a coronary care unit population. South Med J 1988;81:826e9. 48. Moses S, Manji F, Bradley JE, et al. Impact of user fees on attendance at a referral centre for sexually transmitted diseases in Kenya. Lancet 1992;340:463e6. 49. Kelly JA, Murphy DA, Sikkema KJ, et al. Randomised, controlled, community-level HIV-prevention intervention for sexual-risk behaviour among homosexual men in US cities. Community HIV Prevention Research Collaborative. Lancet 1997;350:1500e5. 50. Close J, Ellis M, Hooper R, et al. Prevention of falls in the elderly trial (PROFET): a randomised controlled trial. Lancet 1999;353:93e7. 51. Frasure-Smith N, Lesperance F, Prince RH, et al. Randomised trial of home-based psychosocial nursing intervention for patients recovering from myocardial infarction. Lancet 1997;350:473e9. 52. Oakley A. Experimentation and social interventions: a forgotten but important history. BMJ 1998;317:1239e42. 53. Wellwood J, Johannesen S, Spiegelhalter DJ. How does computer aided diagnosis improve the managment of acute abdominal pain? Ann R Coll Surg Engl 1992;74:40e6. 54. Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials. New York: Springer Verlag, 1998. 55. Friedman CP, Abbas UL. Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems. Int J Med Inform 2003;69:261e72. 56. Black A, Car J, Majeed A, et al. Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research. Inform Prim Care 2008;16:175e7. 57. Car J, Black A, Anandan C, et al. The Impact of Ehealth on the Quality and Safety of Healthcare: A Systematic Overview and Synthesis of the Literature. Birmingham: NHS Connecting for Health Evaluation Programme, 2008. 58. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision support systems on physician perfomance and patient outcomes: a systematic review. JAMA 1998;280:1339e46. 59. Balas EA, Austin SM, Mitchel JA, et al. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med 1996;5:271e8. 60. Johnston ME, Langton KB, Haynes RB, et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome: a critical appraisal of research. Ann Intern Med 1994;120:135e42. 61. Haynes RB, Walker CJ. Computer-aided quality assurance: a critical appraisal of research. Arch Intern Med 1987;147:1297e301. 62. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1223e38. 63. Leung GM, Johnston JM, Tin KYT, et al. Randomised controlled trial of clinical decision tools to improve learning of evidence based medicine in medical students. BMJ 2003;327:1090. 64. Goud R, de Keizer NF, ter Riet G, et al. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabiltation. BMJ 2009;338:b1440. 65. Swartz LHG, Noell JW, Schroeder SW, et al. A randomised control study of a fully automated internet based smoking cessation programme. Tob Control 2006;15:7e12. 66. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005;365:1159e62. 67. Tang JL. Research priorities in traditional Chinese medicine. BMJ 2006;333:391e4. 68. Deeks JJ, Dinnes J, D’Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:iiiex, 1e173. 69. Adams ID, Chan M, Clifford PC, et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ (Clin Res Ed) 1986;293:800e4. 70. Murray E, Burns J, See ST, et al. Interactive health communication applications for people with chronic disease. Cochrane Database Syst Rev 2005;(4):CD004274. 71. Clark RA, Inglis SC, McAlister FA, et al. Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis. BMJ 2007;334:942e5. 72. Guyatt G, Mills EJ, Elbourne D. In the era of systematic reviews, does the size of an individual trial still matter? PLoS Med 2008;5:e4. 73. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010;10:12. 74. Greenhalgh T, Potts HW, Wong G, et al. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q 2009;87:729e88. 75. Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak 2010;10:67. 76. Shepperd S, Lewin S, Straus S, et al. Can we systematically review studies that evaluate complex interventions? PLoS Med 2009;6:e10000086. 77. Pawson R. Evidence-based policy: the promise of ‘realist synthesis.’ Evaluation 2002;8:340e58. 78. Pawson R, Greenhalgh T, Harvey G, et al. Realist reviewea new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005;(10 Suppl 1):21e34. 79. Greenhalgh T, Stones R. Theorising big IT programmes in healthcare: strong structuration theory meets actor-network theory. Soc Sci Med 2010;70:1285e94. 80. Edwards P, Arango M, Balica L, et al; CRASH trial collaborators. Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroids in adults with head injuryeoutcomes at 6 months. Lancet 2005;365:1957e9. 81. Sullivan FM, Swan IRC, Donnan PT, et al. Early treatment with prednisolone or acyclovir in Bell’s Palsy. N Engl J Med 2007;357:1598e607. 82. Altman D, Carroli G, Duley L, et al; The Magpie Trial Collaborative Group. Do women with pre-eclampsia, and theirr babies, benefit from magnesium sulphate? The Magpie Trial: a randomised placebo-controlled trial. Lancet 2002;359:1877e90. 83. Hennekens CH, Buring JE, Manson JE, et al. Lack of effect of long-term supplementation with beta carotene on the incidence of malignant neoplasms and cardiovascular disease. N Engl J Med 1996;334:1145e9. 84. Karsh BT, Weinger MB, Abbott PA, et al. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010;17:617e23. 85. Rigby M, Forsstrom J, Roberts R, et al. Verifying quality and safety in health informatics services. BMJ 2001;323:552e6. 86. Roberts JM, Copeland KL. Clinical websites are currently dangerous to health. Int J Med Inform 2001;62:181e7. 87. Donawa M. FDA final guidance on software validation. Med Device Technol 2002;13:20e4. 88. McDonald CJ. Computerization can create safety hazards: a bar-coding near miss. Ann Intern Med 2006;144:510e16. 89. Newton RC, Mytton OT, Aggarwal R, et al. Making existing technology safer in healthcare. Qual Saf Health Care 2010;19(Suppl 2):i15e24. 90. Robertson A, Cresswell K, Takian A, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ 2010;341: c4564. 91. Ong SG, Liu J, Wong CM, et al. Studies on the respiratory health of primary school children in urban communities of Hong Kong. Sci Total Environ 1991;106:121e35. 92. Peters J, Hedley AJ, Wong CM, et al. Effects of an ambient air pollution intervention and environmental tobacco smoke on children’s respiratory health in Hong Kong. Int J Epidemiol 1996;25:821e8. 93. Hedley AJ, Wong CM, Thach TQ, et al. Cardiorespiratory and all-cause mortality after restriction on sulphur content of fuel in Hong Kong: an intervention study. Lancet 2002;361:787e8. 94. Ho LM, McGhee SM, Hedley AJ, et al. The application of a computerized problemoriented medical record system and its impact on patient care. Int J Med Inform 1999;55:47e59. 95. Contopoulos-Ioannidis DG, Ntzani EE, Ioannidis JP. Translation of highly promising basic science research into clinical applications. Am J Med 2003;114:477e84. 96. Crowley WF Jr. Translation of basic research into useful treatments: how often does it occur? Am J Med 2003;114:503e5. 97. Mill JS. System of Logic, Ratiocinative and Inductive. 5th edn. London: Parker, Son and Bowin, 1862. 98. Copi IM, Cohen C. Introduction to Logic. 13th edn. Englewood Cliffs, NJ: Prentice- Hall, 2008. 99. Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ 2001;322:355e7. 100. Puffer S, Torgerson DJ, Watson J. Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ 2003;327:785e9. 101. Gilbody S, Bower P, Torgerson D, et al. Cluster randomized trials produced similar results to individually randomized trials in a meta-analysis of enhanced care for depression. J Clin Epidemiol 2008;61:160e8. 102. Bennett S, Braunholtz D, Chinn S, et al. Cluster Randomised Trials: Methodological and Ethical Considerations. MRC Clinical Trials Series. London: Medical Research Council, 2002. 103. Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325:941e4. 104. Eminovic N, de Keizer NF, Wyatt JC, et al. Teledermatologic consultation and reduction in referrals to dermatologists: a cluster randomized controlled trial. Arch Dermatol 2009;145:558e64. 105. Friedman B. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141e5. 106. Fries JF, Krishnan E. Equipoise, design bias and randomized controlled trials: the elusive ethics of new drug development. Arthritis Res Ther 2004;6:R250e5. 107. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006;6:54. 108. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 2007;28:182e91. 109. Grimshaw J, Campbell M, Eccles M, et al. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract 2000;(17 Suppl 1):S11e18. 110. Carey TS, Garrett J. Patterns of ordering diagnostic tests for patients with acute low back pain. The North Carolina Back Pain Project. Ann Intern Med 1996;125:807e14. 111. Jones TV, Gerrity MS, Earp J. Written case simulations: do they predict physicians’ behaviour? J Clin Epidemiol 1990;43:805e15. 112. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via a computer charting system: effects on the care of febrile children less than three years of age. J Am Med Inform Assoc 2000;7:186e95. 113. Lee KL, Pryor DB, Harrell FE Jr, et al. Predicting outcome in coronary disease: statistical models versus expert clinicians. Am J Med 1986;80:553e60. 114. Hlatky MA, Califf RM, Harrell FE Jr, et al. Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery. J Am Coll Cardiol 1988;11:237e45. 115. Vall Mayans M, Hall AJ, Inskip HM, et al. Do bedbugs transmit hepatitis B?. Lancet 1994;343:761e3. 180
URI:	http://wrap.warwick.ac.uk/id/eprint/41593

Data sourced from Thomson Reuters' Web of Knowledge

Request changes to a record

Actions (login required)

View Item