Royle, Pamela and Waugh, Norman. (2003) Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Clinical Excellence appraisal system. Health Technology Assessment, Vol.7 (No.34). ISSN 1366-5278

Preview	Text WRAP_Waugh_lit searchTAR.pdf - Published Version Download (271Kb) | Preview
Preview	Text (Cover sheet) WRAP_waugh8_Coversheet_FinalVersion.pdf Download (168Kb) | Preview

Abstract

Background In the UK, one part of the remit of the National Institute for Clinical Excellence (NICE) is to carry out a programme of technology appraisals. These are done to a fairly tight timetable in order not to delay the guidance on new technologies. Each appraisal is underpinned by a Technology Assessment Report (TAR) commissioned from a group of academic units. As the TAR process is relatively new, and is still evolving, the methods used for its literature searching have been largely based on the well-established and documented methods used for Cochrane reviews. These involve comprehensive searching of a variety of sources to protect against bias, but can add substantially to the time and costs of carrying out a review. However, resource constraints require that TARs are produced as efficiently as possible, and to a tight timetable, which means that not all of the Cochrane methods can be applied, or are appropriate. In addition, it is not known whether the marginal benefits of exhaustive searching justify the costs. The challenge for those undertaking TARs is to know how best to adapt and optimise, and extend when necessary, the Cochrane-based search strategies, so that searching can be done both rapidly and systematically. Objective To contribute to making searching for TARs more cost-effective by suggesting an optimum literature retrieval strategy, based on empirical data obtained from a sample of recent TARs, which balances comprehensiveness and efficiency. Methods A sample of 20 recent TARs was studied. All sources used to search for clinical and cost-effectiveness studies were recorded. In addition, all studies that were included in the clinical and cost-effectiveness sections of the TARs were identified, and their characteristics recorded, including author, journal, year, study design, study size and quality score. Each was also classified by publication type, and then checked to see whether it was indexed in the following databases: MEDLINE, EMBASE, and then either the Cochrane Controlled Trials Register (CCTR) for clinical effectiveness studies or the NHS Economic Evaluation Database (NHS EED) for the cost-effectiveness studies. Any study not found in at least one of these databases was checked to see whether it was indexed in the Science Citation Index (SCI) and BIOSIS, and the American Society of Clinical Oncology (ASCO) Online if a cancer review. Any studies still not found were investigated further to see whether they were in a number of additional databases. Results Sources searched The median number of sources searched per TAR was 20, and the range was from 13 to 33 sources. Six sources (CCTR, DARE, EMBASE, MEDLINE, NHS EED and sponsor/industry submissions to NICE) were used in all reviews. Clinical effectiveness studies There were 424 studies in total. The publication types were: published 80%, meeting abstracts 11.3% and unpublished 8.7%. Eighty per cent of reviews included at least one abstract or unpublished study (60% included at least one abstract and 50% included at least one unpublished study). The median number of studies included per TAR was 19.5 (range 2–41). The median number of participants included per TAR was 2787 (range 69–97,570). Evidence from non-randomised controlled trial (RCT) studies was used in 45% of TARs. The proportion of studies classified either as published in full or as abstracts, and found indexed in the following databases, was: MEDLINE 82.7%, EMBASE 78.6% and CCTR 50.1%. The cumulative percentage of studies found after searching these three databases was 87.3%. Adding SCI, BIOSIS and ASCO Online increased this to 98.2%. Eighty-seven per cent of studies were indexed in both MEDLINE and EMBASE. Cost-effectiveness studies The 130 studies were classified as: published 73.1%, unpublished 23.8%, abstracts 1.5% and grey literature 1.5%. The median number of studies used was 4.0. The percentage of studies classified as either published in full or as abstracts, and found indexed in the following databases, was: MEDLINE 86.6%, EMBASE 86.6% and NHS EED 40.2%. The cumulative percentage of these studies found indexed after searching the three databases was 94.8%. Adding SCI and ASCO Online increased this to 97.9%. Studies used in the economic modelling The 121 articles were classified as: published 50.4%, abstracts 5.0%, reference sources 17.4%, unpublished 17.4% and grey literature 9.8%. The median number of studies used for the 14 TARs that included an economic model was 9.0 per TAR. Search terms for identifying non-RCTs A sensitive search filter, constructed for MEDLINE and using the search terms from the bibliographic records in the included studies, retrieved only 85% of the known sample. Therefore, it is recommended that when searching for non-RCT studies a search is done for the intervention alone, and records are then scanned manually for those that look relevant. Conclusions Searching additional databases beyond the Cochrane Library (which includes CCTR, NHS EED and the HTA database), MEDLINE, EMBASE and SCI, plus BIOSIS limited to meeting abstracts only, is seldom effective in retrieving additional studies for inclusion in the clinical and cost-effectiveness sections of TARs (apart from reviews of cancer therapies, where a search of the ASCO database is recommended). A more selective approach to database searching would suffice in most cases and would save resources, thereby making the TAR process more efficient. However, searching non-database sources (including submissions from manufacturers, recent meeting abstracts, contact with experts and checking reference lists) does appear to be a productive way of identifying further studies.

Item Type:	Journal Article
Subjects:	R Medicine > R Medicine (General)
Divisions:	Faculty of Medicine > Warwick Medical School > Health Sciences
Library of Congress Subject Headings (LCSH):	Information storage and retrieval systems -- Medicine, Medicine -- Research, Systematic reviews (Medical research)
Journal or Publication Title:	Health Technology Assessment
Publisher:	National Coordinating Centre for Health Technology Assessment
ISSN:	1366-5278
Date:	2003
Volume:	Vol.7
Number:	No.34
Number of Pages:	64
Status:	Peer Reviewed
Publication Status:	Published
Funder:	NIHR Health Technology Assessment Programme (Great Britain)
References:	1. Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987;106:485–8. 2. Clarke M, Oxman AD, editors. Cochrane Reviewer's Handbook 4.1.6 (updated January 2003). http://www.cochrane.dk/cochrane/handbook/handbo ok.htm. Accessed 23 April 2003. 3. Glasziou P, Irwig L, Bain C, Colditz G. Systematic reviews in health care: a practical guide. Cambridge: Cambridge University Press; 2001. 4. Cochrane Collaboration. http://www.cochrane.de 5. Antes G, Oxman AD. The Cochrane Collaboration in the 20th century. In Egger M, Davey SG, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Books; 2001. pp. 447–73. 6. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998;280:278–80. 7. National Institute for Clinical Excellence. http://www.nice.org.uk 8. Best L, Stevens A, Colin Jones D. Rapid and responsive health technology assessment: the development and evaluation process in the South and West region of England. Journal Clinical Efficiency 1997;2:51–6. 9. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003;7:1–76. 10. Egger M, Davey SG, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Books; 2001. 11. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000;4(10):1–115. 12. Egger M, Dickersin K, Davey SG. Problems and limitations in conducting systematic reviews. In Egger M, Davey SG, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Books; 2001. pp. 43–68. 13. Campanario JM. Consolation for the scientist – sometimes it is hard to publish papers that are later highly-cited. Social Studies of Science 1993;23:342–62. 14. Spier RE. Peer review and innovation. Sci Eng Ethics 2002;8:99–108. 15. Pittler MH, Abbot NC, Harkness EF, Ernst E. Location bias in controlled clinical trials of complementary/alternative therapies. J Clin Epidemiol 2000;53:485–9. 16. Lee KP, Schotland M, Bacchetti P, Bero LA. Association of journal quality indicators with methodological quality of clinical research articles. JAMA 2002;287:2805–8. 17. Hill DR, Stickell HN. Brandon/Hill selected list of print books and journals for the small medical library. Bulletin of the Medical Libraries Association 2001;89:131–53. 18. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions (Cochrane Methodology Review). In The Cochrane Library (Issue 1). Oxford: Update Software; 2003. 19. Scherer RW, Langenberg P. Full publication of results initially presented in abstracts (Cochrane Methodology Review). Cochrane Library. Oxford: Update Software; 2001. 20. MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG. How useful are unpublished data from the Food and Drug Administration in metaanalysis? J Clin Epidemiol 2003;56:44–51. 21. Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, et al. Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993; 269:2749-53. 22. Jefferson TO, Alderson P, Davidoff F, Wager E. Editorial peer-review for improving the quality of reports of biomedical studies (Cochrane Methodology Review). In The Cochrane Library (Issue 1). Oxford: Update Software; 2003. 23. Savoie I, Helmer D, Green CJ, Kazanjian A. Beyond Medline: reducing bias through extended systematic review search. Int J Technol Assess Health Care 2003;19:168–78. 24. Sampson M, Campbell K, Ajiferuke I, Moher D. Randomized controlled trials in pediatric complementary and alternative medicine: where can they be found? BMC Pediatr 2003;3:1. 25. Avenell A, Handoll HHG, Grant AM. Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr 2001;73:505–10. 26. Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials 2000;21:476–87. 27. Royle P, Milne R. Literature searching for randomised controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care 2003; in press. 28. Cook AM, Finlay IG, Edwards AG, Hood K, Higginson IJ, Goodwin DM, et al. Efficiency of searching the grey literature in palliative care. J Pain Symptom Manage 2001;22:797–801. 29. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1–12. 30. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ 1996;313:275–83. 31. Lefebvre C, Clarke MJ. Identifying randomised trials. In Egger M, Davey SG, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. London: BMJ Books; 2001. pp. 69–86. 32. Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? Int J Epidemiol 2002;31:1–5. 33. NICE. Compilation – summary of guidance issued to the NHS in England and Wales. London: National Institute for Clinical Excellence; October 2002 (Issue 5). 34. Juni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in metaanalyses of controlled trials: empirical study. Int J Epidemiol 2002;31:115–23. 35. NHS Centre for Reviews and Dissemination. Centre for Reviews and Dissemination (CRD). Improving access to cost-effectiveness information for health care decision making: the NHS Economic Evaluation Database. CRD Report 6. 2nd ed. York: University of York; 2001. 36. Eady A. Secondary publications: economic analyses. In McKibbon A, editor. PDQ evidence-based principles and practice. Hamilton: Decker; 1999. pp. 173–86. 37. Sassi F, Archard L, McDaid D. Searching literature databases for health care economic evaluations: how systematic can we afford to be? Med Care 2002; 40:387–94. 38. Clarke M, Oxman AD, editors. Locating and selecting studies. Cochrane Reviewers Handbook 4.1.6 (updated January 2003) Section 5. In The Cochrane Library (Issue 2). Oxford: Update Software; 2003. 39. Topfer LA, Parada A, Menon D, Noorani H, Perras C, Serra-Prat M. Comparison of literature searches on quality and costs for health technology assessment using the MEDLINE and EMBASE databases. Int J Technol Assess Health Care 1999; 15:297–303. 40. Royle P. Literature searching for randomised controlled trials used in Cochrane reviews: rapid versus exhaustive searches [dissertation]. Southampton: University of Southampton; 2001. 41. Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials (Cochrane Methodology Review). In The Cochrane Library (Issue 1). Oxford: Update Software; 2003.
URI:	http://wrap.warwick.ac.uk/id/eprint/52133

Request changes to a record

Actions (login required)

View Item