1. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry
systems in facilitating medication errors. JAMA 2005;293:1197e203.
2. Tierney WM, Overhage JM, McDonald CJ. A plea for controlled trials in medical
informatics. J Am Med Inform Assoc 1994;1:353e5.
3. Heathfield HA, Buchan IB. Current evaluations of information technology in health
care are often inadequate. BMJ 1996;313:1008.
4. Heathfield HA, Pitty D, Hanka R. Evaluating information technology in health care:
barriers and challenges. BMJ 1998;316:1959e61.
5. Kaplan B. Evaluating informatics applicationsdsome alternative approaches, and
call for methodological pluralism. Int J Med Inform 2001;64:39e56.
6. UK Institute of Health Informatics. Evaluation of Electronic Patient and Health
Record Projects. Winchester: ERDIP Programme, NHS Information Authority, 2001.
7. Stoop A, Heathfield H, de Mul M, et al. Evaluation of patient care information
systems: theory and practice. In: Berg M, ed. Health Information Management:
Integrating Information Technology in Health Care Work. London: Routledge,
2004:206e29.
8. Brender J. Handbook of Evaluation Methods for Health Informatics. London:
Elsevier Academic Press, 2006.
9. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous
systematic evaluation. PLoS Med 2009;6:e10000126.
10. Rigby M. Health informatics as a tool to improve quality in non-acute carednew
opportunities and a matching need for a new evaluation paradigm. Int J Med Inform
1999;56:141e50.
11. Patel VL, Kaufmann DR, Arocha JF. Emerging paradigms of cognition in medical
decision making. J Biomed Inform 2002;35:52e75.
12. Kaplan B, Shaw NT. People, Organizational, and Social Issues. IMIA Yearbook of
Medical Informatics. Heidelberg: Schattauer, 2002:91e102.
13. Beale S, Sanderson D, Kruger J; for the York Health Economics Consortium.
Evaluation of the Telecare Development Programme: Final Report. Edinburgh:
Scottish Government, 2009.
14. Friedman CP, Wyatt JC. Evaluation Methods in Bio-Medical Informatics. 2nd edn.
New York: Springer-Verlag, 2006.
15. Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall,
1991.
16. Pope C, Mays N. Qualitative Research in Health Care. 2nd edn. London: BMJ
Publishing, 2000.
17. Rothman KJ, Greenland S. Modern Epidemiology. 2nd edn. Philadelphia:
Lippincott-Raven, 1998. 18. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of
quasi-experimental studies in medical informatics. J Am Med Inform Assoc
2006;13:16e23.
19. de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how
did the quality of healthcare IT evaluation publications develop from 1982 to 2005?
Int J Med Inform 2008;77:41e9.
20. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of
computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am
Med Inform Assoc 2009;16:531e8.
21. Liu JLY, Wyatt JC, Deeks JJ, et al. Systematic reviews of clinical decision tools for
acute abdominal pain. Health Technol Assess 2006;10:1e186, iiieiv.
22. Copi IM, Cohen C. Introduction to Logic. 9th edn. Englewood Cliffs, NJ: Prentice-
Hall, 1994.
23. Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J
Med Inform 2003;69:251e9.
24. Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction
to qualitative methods in health and health services. BMJ 1995;311:42e5.
25. Drummond MF, Sculpher MJ, Torrance GW, et al. Methods for the Economic
Evaluation of Health Care Programmes. 3rd edn. New York: Oxford University Press,
2005.
26. Pocock SJ. Clinical Trials: A Practical Approach. Chichester: John Wiley and Sons,
1983.
27. Schulz KF, Altman DG, Moher D; the CONSORT Group. CONSORT 2010 Statement:
updated guidelines for reporting parallel-group randomized trials. BMJ 2010;340:c332.
28. Moher D, Hopewell S, Schulz KF, et al. The CONSORT Statement 2010 updated
guidelines for reporting randomized trials: explanation and elaboration. BMJ
2010;340:c869.
29. Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians’ diagnostic
reasoning by computer-based consultations: a multisite study of 2 systems. JAMA
1999;282:1851e6.
30. Macintyre S, Petticrew M. Good intentions and received wisdom are not enough.
J Epidemiol Community Health 2000;54:802e3.
31. Wynder EL. Studies in mechanism and prevention: striking a proper balance. Am J
Epidemiol 1994;139:547e9.
32. McLean S, Chandler D, Nurmatov U, et al. Telehealthcare for asthma. Cochrane
Database Syst Rev 2010;(10):CD007717.
33. Liu JLY, Wyatt JC. The Impact of Telepsychiatry, Teledermatology, Teleent and
Teleradiology on the Health Services: A Rapid Review. Portland, OR: Medicaid
Evidence-Based Decisions Program, 2008.
34. Clark JS, Mair FS, O’Donnell C, et al. E-health: implementation and
evaluation research in Scotlandda scoping exercise. J Telemed Telecare
2008;14:119e21.
35. Abraham EP, Chain E, Fletcher CM. Further observations on penicillin. Lancet
1941;2:177e88.
36. Peto R, Pike MC, Armitage P, et al. Design and analysis of randomized clinical trials
requiring prolonged observation of each patient. I. Introduction and design. Br J
Cancer 1976;34:585e612.
37. Collins R, Peto R, Gray R, et al. Large-Scale Randomized Evidence: Trials and
Overviews. Oxford Textbook of Medicine. Oxford: Oxford University Press,
1998:21e32.
38. Peto R, Baigent C. Trials: the next 50 years. Large scale randomised evidence of
moderate benefits. BMJ 1998;317:1170e1.
39. Collins R, MacMahon S. Reliable assessment of the effects of treatment on
mortality and major morbidity, I: clinical trials. Lancet 2001;357:373e80.
40. Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for
clinical trials. Am J Med 1982;72:233e40.
41. Tang JL, Leung PC. An efficacy-driven approach to the research and development
of traditional Chinese medicine. Hong Kong Med J 2001;7:375e80.
42. Anon. Effectiveness of health checks conducted by nurses in primary care: final
results of the OXCHECK study. Imperial Cancer Research Fund OXCHECK Study
Group. BMJ 1995;310:1099e104.
43. Proudfoot J, Guest D, Carson J, et al. Effect of cognitive-behavioural training on
job-finding among long-term unemployed people. Lancet 1997;350:96e100.
44. Wyatt JC, Paterson-Brown S, Johanson R, et al. Trial of outreach visits to enhance
the use of systematic reviews in 25 obstetric units. BMJ 1998;317:1041e6.
45. Hobbs M, Mayou R, Harrison B, et al. A randomised controlled trial of psychological
debriefing for victims of road traffic accidents. BMJ 1996;313:1438e9.
46. Allen K, Blascovich J. The value of service dogs for people with severe ambulatory
disabilities. A randomised controlled trial. JAMA 1996;275:1001e6.
47. Byrd RC. Positive therapeutic effects of intercessory prayer in a coronary care unit
population. South Med J 1988;81:826e9.
48. Moses S, Manji F, Bradley JE, et al. Impact of user fees on attendance at a referral
centre for sexually transmitted diseases in Kenya. Lancet 1992;340:463e6.
49. Kelly JA, Murphy DA, Sikkema KJ, et al. Randomised, controlled, community-level
HIV-prevention intervention for sexual-risk behaviour among homosexual men in US
cities. Community HIV Prevention Research Collaborative. Lancet
1997;350:1500e5.
50. Close J, Ellis M, Hooper R, et al. Prevention of falls in the elderly trial (PROFET):
a randomised controlled trial. Lancet 1999;353:93e7.
51. Frasure-Smith N, Lesperance F, Prince RH, et al. Randomised trial of home-based
psychosocial nursing intervention for patients recovering from myocardial infarction.
Lancet 1997;350:473e9.
52. Oakley A. Experimentation and social interventions: a forgotten but important
history. BMJ 1998;317:1239e42.
53. Wellwood J, Johannesen S, Spiegelhalter DJ. How does computer aided
diagnosis improve the managment of acute abdominal pain? Ann R Coll Surg Engl
1992;74:40e6.
54. Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials. New York:
Springer Verlag, 1998.
55. Friedman CP, Abbas UL. Is medical informatics a mature science? A review of
measurement practice in outcome studies of clinical systems. Int J Med Inform
2003;69:261e72.
56. Black A, Car J, Majeed A, et al. Strategic considerations for improving the quality
of eHealth research: we need to improve the quality and capacity of academia to
undertake informatics research. Inform Prim Care 2008;16:175e7.
57. Car J, Black A, Anandan C, et al. The Impact of Ehealth on the Quality and Safety of
Healthcare: A Systematic Overview and Synthesis of the Literature. Birmingham:
NHS Connecting for Health Evaluation Programme, 2008.
58. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision
support systems on physician perfomance and patient outcomes: a systematic
review. JAMA 1998;280:1339e46.
59. Balas EA, Austin SM, Mitchel JA, et al. The clinical value of computerized
information services. A review of 98 randomized clinical trials. Arch Fam Med
1996;5:271e8.
60. Johnston ME, Langton KB, Haynes RB, et al. Effects of computer-based clinical
decision support systems on clinician performance and patient outcome: a critical
appraisal of research. Ann Intern Med 1994;120:135e42.
61. Haynes RB, Walker CJ. Computer-aided quality assurance: a critical appraisal of
research. Arch Intern Med 1987;147:1297e301.
62. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision
support systems on practitioner performance and patient outcomes: a systematic
review. JAMA 2005;293:1223e38.
63. Leung GM, Johnston JM, Tin KYT, et al. Randomised controlled trial of clinical
decision tools to improve learning of evidence based medicine in medical students.
BMJ 2003;327:1090.
64. Goud R, de Keizer NF, ter Riet G, et al. Effect of guideline based computerised
decision support on decision making of multidisciplinary teams: cluster randomised
trial in cardiac rehabiltation. BMJ 2009;338:b1440.
65. Swartz LHG, Noell JW, Schroeder SW, et al. A randomised control study of a fully
automated internet based smoking cessation programme. Tob Control
2006;15:7e12.
66. Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in
PubMed journals. Lancet 2005;365:1159e62.
67. Tang JL. Research priorities in traditional Chinese medicine. BMJ
2006;333:391e4.
68. Deeks JJ, Dinnes J, D’Amico R, et al. Evaluating non-randomised intervention
studies. Health Technol Assess 2003;7:iiiex, 1e173.
69. Adams ID, Chan M, Clifford PC, et al. Computer aided diagnosis of acute abdominal
pain: a multicentre study. BMJ (Clin Res Ed) 1986;293:800e4.
70. Murray E, Burns J, See ST, et al. Interactive health communication applications for
people with chronic disease. Cochrane Database Syst Rev 2005;(4):CD004274.
71. Clark RA, Inglis SC, McAlister FA, et al. Telemonitoring or structured telephone
support programmes for patients with chronic heart failure: systematic review and
meta-analysis. BMJ 2007;334:942e5.
72. Guyatt G, Mills EJ, Elbourne D. In the era of systematic reviews, does the size of
an individual trial still matter? PLoS Med 2008;5:e4.
73. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist
review of what works, for whom and in what circumstances. BMC Med Educ
2010;10:12.
74. Greenhalgh T, Potts HW, Wong G, et al. Tensions and paradoxes in electronic
patient record research: a systematic literature review using the meta-narrative
method. Milbank Q 2009;87:729e88.
75. Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in
understanding the implementation of information technology developments in
healthcare. BMC Med Inform Decis Mak 2010;10:67.
76. Shepperd S, Lewin S, Straus S, et al. Can we systematically review studies that
evaluate complex interventions? PLoS Med 2009;6:e10000086.
77. Pawson R. Evidence-based policy: the promise of ‘realist synthesis.’ Evaluation
2002;8:340e58.
78. Pawson R, Greenhalgh T, Harvey G, et al. Realist reviewea new method of
systematic review designed for complex policy interventions. J Health Serv Res
Policy 2005;(10 Suppl 1):21e34.
79. Greenhalgh T, Stones R. Theorising big IT programmes in healthcare:
strong structuration theory meets actor-network theory. Soc Sci Med
2010;70:1285e94.
80. Edwards P, Arango M, Balica L, et al; CRASH trial collaborators. Final results of
MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroids in
adults with head injuryeoutcomes at 6 months. Lancet 2005;365:1957e9.
81. Sullivan FM, Swan IRC, Donnan PT, et al. Early treatment with prednisolone or
acyclovir in Bell’s Palsy. N Engl J Med 2007;357:1598e607.
82. Altman D, Carroli G, Duley L, et al; The Magpie Trial Collaborative Group. Do
women with pre-eclampsia, and theirr babies, benefit from magnesium sulphate?
The Magpie Trial: a randomised placebo-controlled trial. Lancet
2002;359:1877e90. 83. Hennekens CH, Buring JE, Manson JE, et al. Lack of effect of long-term
supplementation with beta carotene on the incidence of malignant neoplasms and
cardiovascular disease. N Engl J Med 1996;334:1145e9.
84. Karsh BT, Weinger MB, Abbott PA, et al. Health information technology: fallacies
and sober realities. J Am Med Inform Assoc 2010;17:617e23.
85. Rigby M, Forsstrom J, Roberts R, et al. Verifying quality and safety in health
informatics services. BMJ 2001;323:552e6.
86. Roberts JM, Copeland KL. Clinical websites are currently dangerous to health. Int J
Med Inform 2001;62:181e7.
87. Donawa M. FDA final guidance on software validation. Med Device Technol
2002;13:20e4.
88. McDonald CJ. Computerization can create safety hazards: a bar-coding near miss.
Ann Intern Med 2006;144:510e16.
89. Newton RC, Mytton OT, Aggarwal R, et al. Making existing technology safer in
healthcare. Qual Saf Health Care 2010;19(Suppl 2):i15e24.
90. Robertson A, Cresswell K, Takian A, et al. Implementation and adoption of
nationwide electronic health records in secondary care in England: qualitative
analysis of interim results from a prospective national evaluation. BMJ 2010;341:
c4564.
91. Ong SG, Liu J, Wong CM, et al. Studies on the respiratory health of primary
school children in urban communities of Hong Kong. Sci Total Environ
1991;106:121e35.
92. Peters J, Hedley AJ, Wong CM, et al. Effects of an ambient air pollution
intervention and environmental tobacco smoke on children’s respiratory health in
Hong Kong. Int J Epidemiol 1996;25:821e8.
93. Hedley AJ, Wong CM, Thach TQ, et al. Cardiorespiratory and all-cause mortality
after restriction on sulphur content of fuel in Hong Kong: an intervention study.
Lancet 2002;361:787e8.
94. Ho LM, McGhee SM, Hedley AJ, et al. The application of a computerized problemoriented
medical record system and its impact on patient care. Int J Med Inform
1999;55:47e59.
95. Contopoulos-Ioannidis DG, Ntzani EE, Ioannidis JP. Translation of highly
promising basic science research into clinical applications. Am J Med
2003;114:477e84.
96. Crowley WF Jr. Translation of basic research into useful treatments: how often
does it occur? Am J Med 2003;114:503e5.
97. Mill JS. System of Logic, Ratiocinative and Inductive. 5th edn. London: Parker, Son
and Bowin, 1862.
98. Copi IM, Cohen C. Introduction to Logic. 13th edn. Englewood Cliffs, NJ: Prentice-
Hall, 2008.
99. Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ
2001;322:355e7.
100. Puffer S, Torgerson DJ, Watson J. Evidence for risk of bias in cluster randomised
trials: review of recent trials published in three general medical journals. BMJ
2003;327:785e9.
101. Gilbody S, Bower P, Torgerson D, et al. Cluster randomized trials produced similar
results to individually randomized trials in a meta-analysis of enhanced care for
depression. J Clin Epidemiol 2008;61:160e8.
102. Bennett S, Braunholtz D, Chinn S, et al. Cluster Randomised Trials: Methodological
and Ethical Considerations. MRC Clinical Trials Series. London: Medical Research
Council, 2002.
103. Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based
guidelines on management of asthma and angina in adults in primary care: cluster
randomised controlled trial. BMJ 2002;325:941e4.
104. Eminovic N, de Keizer NF, Wyatt JC, et al. Teledermatologic consultation and
reduction in referrals to dermatologists: a cluster randomized controlled trial. Arch
Dermatol 2009;145:558e64.
105. Friedman B. Equipoise and the ethics of clinical research. N Engl J Med
1987;317:141e5.
106. Fries JF, Krishnan E. Equipoise, design bias and randomized controlled trials: the
elusive ethics of new drug development. Arthritis Res Ther 2004;6:R250e5.
107. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC
Med Res Methodol 2006;6:54.
108. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized
trials. Contemp Clin Trials 2007;28:182e91.
109. Grimshaw J, Campbell M, Eccles M, et al. Experimental and quasi-experimental
designs for evaluating guideline implementation strategies. Fam Pract 2000;(17
Suppl 1):S11e18.
110. Carey TS, Garrett J. Patterns of ordering diagnostic tests for patients with acute
low back pain. The North Carolina Back Pain Project. Ann Intern Med
1996;125:807e14.
111. Jones TV, Gerrity MS, Earp J. Written case simulations: do they predict physicians’
behaviour? J Clin Epidemiol 1990;43:805e15.
112. Schriger DL, Baraff LJ, Buller K, et al. Implementation of clinical guidelines via
a computer charting system: effects on the care of febrile children less than three
years of age. J Am Med Inform Assoc 2000;7:186e95.
113. Lee KL, Pryor DB, Harrell FE Jr, et al. Predicting outcome in coronary
disease: statistical models versus expert clinicians. Am J Med
1986;80:553e60.
114. Hlatky MA, Califf RM, Harrell FE Jr, et al. Comparison of predictions based on
observational data with the results of randomized controlled clinical trials of
coronary artery bypass surgery. J Am Coll Cardiol 1988;11:237e45.
115. Vall Mayans M, Hall AJ, Inskip HM, et al. Do bedbugs transmit hepatitis B?.
Lancet 1994;343:761e3.
180