The Library

Independent verification and validation of an industrial simulation model

Tools

Robinson, Stewart and Brooks, Roger J.. (2010) Independent verification and validation of an industrial simulation model. Simulation : Transactions of the Society for Modeling and Simulation International, Vol.86 (No.7). pp. 405-416. ISSN 0037-5497

Preview

PDF
WRAP_Robinson_0770216-lb-150611-independent_verification_and_validation_final.pdf - Accepted Version - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (247Kb)

Official URL: http://dx.doi.org/10.1177/0037549709341582

Abstract

The independent verification and validation (IV&V) of simulation models is largely restricted to applications in the military and public policy domains. There is little evidence of IV&V for industrial simulation models. This is largely because industrial simulations are normally of a much smaller scale and do not warrant a full IV&V. A procedure for IV&V of industrial simulation models is described that provides a viable alternative where the cost and time of IV&V must be contained. The procedure consists of the following activities: structured walkthrough, review of model assumptions, code examination, review of verification procedures, replications analysis, review of static analysis, review of output reporting, and investigation of results and experimentation. The IV&V of a Sellafield Limited supply chain model is described.

Item Type:

Journal Article

Subjects:

H Social Sciences > HD Industries. Land use. Labor
Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software

Divisions:

Faculty of Social Sciences > Warwick Business School

Library of Congress Subject Headings (LCSH):

Manufacturing processes -- Simulation methods, Computer programs -- Verification, Computer programs -- Validation, Sellafield Ltd.

Journal or Publication Title:

Simulation : Transactions of the Society for Modeling and Simulation International

Publisher:

Sage Publications Inc

ISSN:

0037-5497

Official Date:

July 2010

Dates:

Date	Event
July 2010	Published

Volume:

Vol.86

Number:

No.7

Number of Pages:

Page Range:

pp. 405-416

Identification Number:

10.1177/0037549709341582

Status:

Peer Reviewed

Publication Status:

Published

Access rights to Published version:

Restricted or Subscription Access

References:

1. Davis, P.K. 1992. Generalizing Concepts of Verification, Validation and Accreditation (VV&A) for Military Simulation. R-4249-ACQ, October 1992, RAND, Santa Monica, CA.
2. Carson, J.S. 1986. Convincing users of model's validity is challenging aspect of modeler's job. Industrial Engineering, 18 (6): 74-85.
3. Robinson, S. 1999. Simulation Verification, Validation and Confidence: A Tutorial. Transactions of the Society for Computer Simulation International, 16 (2): 63-69.
4. Gass, S.I. and Joel, L.S. 1981. Concepts of model confidence. Computers and Operations Research, 8 (4): 341-346.
5. Balci, O. 1985. Guidelines for successful simulation studies. Technical Report TR-85-2, Department of Computer Science, Virginia Tech, Blacksburg, VA.
6. Gass, S.I. 1983. Decision-aiding models: validation, assessment, and related issues for policy analysis. Operations Research, 31 (4): 603-631.
7. DoD 1996. DoD modeling and simulation verification, validation, and accreditation. Department of Defense Instruction 5000.61, Apr.
8. Balci, O. 2001. A methodology for certification of modeling and simulation applications. ACM Transaction on Modeling and Computer Simulation, 11 (4): 352-377.
9. Tullos-Banks, H.L., Parker, C.T. and Collins, K.W. 2005. Verification, validation and accreditation of federations. Proceedings of the 2005 Spring Simulation Interoperability Workshop, www.sisostds.org (accessed June 2009).
10. Sargent, R.G. 2007. Verification and validation of simulation models. Proceedings of the 2007 Winter Simulation Conference (Henderson, S.G., Biller, G., Hsieh, M.-H., Shortle, J., Tew, J.D., Barton, R.R., eds.), IEEE, Piscataway, NJ: 124-137.
11. Jacquart, R., Bouc, P. and Brade, D. 2004. A common validation, verification and accreditation framework for simulatons: project JP 11.20, REVVA. Proceedings of the 2004 Spring Simulation Interoperability Workshop, www.sisostds.org (accessed June 2009).
12. Schwartzenburg, F., Oates, W., Park, J., Johnson, D., Stutzman, M., Bailey, M. and Youngblood, S. 2007. Verification, validation, and accreditation (VV&A). One voice – unified, common and cross-cutting. Proceedings of the 2007 Summer Computer Simulation Conference (G.A. Wainer, ed.), Society for Modeling and Simulation International, San Diego, CA: 429-436.
13. Stutzman, M. and Park, J. 2004. Navy modeling and simulation standards for verification, validation, and accreditation. Proceedings of the 2004 Fall Simulation Interoperability Workshop, www.sisostds.org (accessed June 2009).
14. Arthur, J.D. and Nance, R.E. 1996. Independent verification and validation: a missing link in simulation methodology? Proceedings of the 1996 Winter Simulation Conference (Charnes, J.M., Morrice, D.J., Brunner, D.T. and Swain, J.J., eds.). IEEE, Piscataway, NJ: 230-236.
15. Balci, O. 1998. Verification, validation and testing. Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice (Banks, J. ed.), Wiley, New York: 335-393.
16. Robinson, S. 1998. Measuring service quality in the process of delivering a simulation study: the customer’s perspective. International Transactions in Operational Research, 5 (5): 357-374.
17. Ören, T.I. 1981. Concepts and criteria to assess acceptability of simulation studies: a frame of reference. Communications of the ACM, 28 (2): 190-201.
18. Ören, T.I. 1984. Quality assurance in modelling and simulation: a taxonomy. Simulation and Model-based Methodologies: An Integrative Approach (Ören, T.I., Zeigler, B.P. and Elzas, M.S., eds.). Springer-Verlag, Heidelberg, Germany: 477-517.
19. Liu, F., Yang, M. and Zicai, W. 2005. Study on simulation credibility metrics. Proceeding of the 2005 Winter Simulation Conference (M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, eds.). IEEE, Piscataway, NJ: 2554-2560.
20. Boehm, B.W., Brown, J.R., Kaspar, H., Lipow, M., Macleod, G.J. and Merrit, M.J. 1978. Characteristics of Software Quality. North-Holland, Amsterdam.
21. Balci, O. Nance, R.E., Arthur, J.D. and Ormsby, W.F. 2002. Expanding our Horizons in Verification, Validation, and Accreditation Research and Practice. Proceedings of the 2002 Winter Simulation Conference (Conference (E. Yücesan, C.-H. Chen, J.L. Snowden and J.M. Charnes, eds.). IEEE, Piscataway, NJ: 653-663.
22. Balci, O. 2004. Quality assessment, verification, and validation of modeling and simulation applications. Proceeding of the 2004 Winter Simulation Conference (R,G, Ingalls, M.D. Rossetti, J.S. Smith and B.A Peters, eds.). IEEE, Piscataway, NJ: 122-129.
23. Sargent, R.G. 1981. An assessment procedure and a set of criteria for use in the evaluation of computerized models and computer-based modelling tools. RADC-TR-80-409, Rome Air Development Center, Air Force Systems Command, Griffiths Air Force Base, NY.
24. Fossett, C.A., Harrison, D., Weintrob, H. and Gass, S.I. 1991. An assessment procedure for simulation models: a case study. Operations Research, 39 (5): 710-723.
25. Gass, S.I. 1977. A procedure for the evaluation of complex models. Proceedings First International Conference in Mathematical Models, University of Missouri.
26. Gass, S.I. 1993. Model accreditation: a rationale and process for determining a numerical rating. European Journal of Operational Research, 66: 250-258.
27. Saaty, T.L. 1980. The Analytic Hierarchy Process. McGraw-Hill, New York.
28. Pohl, S. and Brade, D. 2005. Using fuzzy multi attribute decision theory in the accreditation of models, simulation , and federations. Proceedings of the 2005 Spring Simulation Interoperability Workshop, www.sisostds.org (accessed June 2009).
29. DMSO 2000. V&V tools. vva.dmso.mil/Ref_Docs/VVTools/vvtools.htm (created November 2000, accessed September 2006)
30. Balci, O, Adams. R.J., Myers, D.S. and Nance, R.E. 2002. A collaborative evaluation environment for credibility assessment of modeling and simulation applications. Proceedings of the 2002 Winter Simulation Conference (Conference (E. Yücesan, C.-H. Chen, J.L. Snowden and J.M. Charnes, eds.). IEEE, Piscataway, NJ: 214-220.
31. Ke, F., Ming, Y. and Zicai, W. 2005.The HITVICE VV&A environment. Proceeding of the 2005 Winter Simulation Conference (M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, eds.). IEEE, Piscataway, NJ: 1220-1227.
32. Robinson, S. 2002. Modes of simulation practice: approaches to business and military simulation. Simulation Practice and Theory, 10: 513-523.
33. Gass, S.I. 1977. Evaluation of complex models. Computers and Operations Research, 4: 27-35.
34. Youngblood, S.M., Pace, D.K., Eirich, P.L, Gregg, D.M. and Coolahan, J.E. 2000. Simulation verification, validation, and accreditation. Johns Hopkins APL Technical Digest, 21 (3): 359-367.
35. Page, E.H., Canova, B.S. and Tufarolo, J.A. 1997. A case study of verification, validation, and accreditation for advanced distributed simulation. ACM Transactions on Modeling and Computer Simulation, 7 (3): 393-424.
36. Cochran, J.K., Mackulak, G.T. and Savory, P.A. 1995. Simulation project characteristics in industrial settings. Interfaces, 25 (4): 104-113.
37. Diener, D.A., Hicks, H.R. and Long, L.L. 1992. Comparison of models: ex post facto validation/acceptance? Proceedings of the 1992 Winter Simulation Conference (Swain, J.J., Goldsman, D., Crain, R.C. and Wilson, J.R., eds.). IEEE, Piscataway, NJ: 1095-1103.
38. Robinson, S. and Pidd, M. 1998. Provider and customer expectations of successful simulation projects. Journal of the Operational Research Society, 49 (3): 200-209.
39. Robinson, S. 2004. Simulation: The Practice of Model Development and Use. Wiley, Chichester, UK.
40. Wall, S. and Shaw, I. 2002. Retrieval of intermediate-level waste at Trawsfynydd nuclear power station. Nuclear Energy: Journal of the British Nuclear Energy Society, 41 (6): 391-396.
41. Lanner (2006). Witness 2006. www.lanner.com (accessed June 2009)
42. Landry, M., Malouin, J.L. and Oral, M. 1983. Model validation in operations research. European Journal of Operational Research, 14 (3): 207-220.
43. Sargent, R.G. 1982. Verification and validation of simulation models. Progress in Modelling and Simulation (Cellier, F.E., ed.). Academic Press, London: 159-169.