This content is not included in
your SAE MOBILUS subscription, or you are not logged in.
A Paradigm of Model Validation and Validated Models for Best-Estimate-Plus-Uncertainty Predictions in Systems Engineering
1
Technical Paper
2007-01-1746
ISSN: 0148-7191, e-ISSN: 2688-3627
Annotation ability available
Sector:
Language:
English
Abstract
What constitutes a validated model? What are the criteria that allow one to defensibly make the claim that they are using a validated model in an analysis? These questions get to the heart of what model validation really implies (conceptually, operationally, interpretationally, etc.), and these details are currently the subject of substantial debate in the V&V community. This is perhaps because many contemporary paradigms of model validation have a limited modeling scope in mind, so the validation paradigms do not span different modeling regimes and purposes that are important in engineering. This paper discusses the different modeling regimes and purposes that it is important for a validation theory to span, and then proposes a validation paradigm that appears to span them. The author's criterion for validated models proceeds from a desire to meet an end objective of “best estimate plus uncertainty” (BEPU) in model predictions. Starting from this end, the author works back to the implications on the model validation process (conceptually, operationally, interpretationally, etc.). Ultimately a shift is required in the conceptualization and articulation of model validation, away from contemporary paradigms. Thus, this paper points out weaknesses in contemporary model validation perspectives and proposes a conception of model validation and validated models that seems to reconcile many of the issues.
Recommended Content
Topic
Citation
Romero, V., "A Paradigm of Model Validation and Validated Models for Best-Estimate-Plus-Uncertainty Predictions in Systems EngineeringAlso In
Reliability and Robust Design in Automotive Engineering, 2007
Number: SP-2119; Published: 2007-04-16
Number: SP-2119; Published: 2007-04-16
References
- AIAA 1998 Guide for the Verification and Validation of Computational Fluid Dynamics Simulations American Institute of Aeronautics and Astonautics Reston, Va.
- ASME 2006 Guide for Verification and Validation in Computational Solid Mechanics http://catalog.asme.org/Codes/PrintBook/VV_10_2006_Guide_Verification.cfm
- Ben-Haim, Y. Information Gap Decision Theory-Decisions under severe uncertainty Academic Press London 2001
- Black, A.R. Hobbs M.L. Dowding K.J. Blanchat T.K. “Uncertainty Quantification and Model Validation of Fire/Thermal Response Predictions,” to be presented at the 18th AIAA Computational Fluid Dynamics Conference 25-28 June 2007 Miami, FL
- Carter, J.N. Ballester P.J. Tavassoli Z. King P.R. “Our Calibrated Model has No Predictive Value: An Example from the Petroleum Industry,” presented at the 4th International Conference on Sensitivity Analysis of Model Output (SAMO) Conference Santa Fe, NM March 8-11 2004 http://library.lanl.gov/ccw/samo2004/
- Coleman, H. W. Steele, W. G. Jr. 1989 Experimentation and Uncertainty Analysis for Engineers John Wiley & Sons New York
- Coleman, H.W. Stern, F. “Uncertainties in CFD Code Validation,” Journal of Fluids Engineering Dec. 1997 119 795 803
- Department of Defense 2006 Key Concepts of Verification, Verification, & Accreditation Sept. 15 2006
- Dowding, K.J. Hills R.G. Leslie I.H. Pilch M. Rutherford B.M. Hobbs M.L. “Case Study for Model Validation: Assessing a Model for Thermal Decomposition of Polyurethane Foam,” Sandia National Laboratories report SAND2004-3632 September 2004
- Easterling, R. G. “Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations,” Sandia National Laboratores report, SAND2001-0243 February 2001
- Easterling, R. G. “Measuring Predictive Capability of Computational Models: Foam Degradation Case Study,” Foundations for V&V (Computer Model Verification and Validation) in the 21st Century Workshop October 22-23 2002
- Easterling, R. G. Berger, J. O. “Statistical Foundations for the Validation of Computer Models,” Foundations for V&V (Computer Model Verification and Validation) in the 21st Century Workshop October 22-23 2002
- Hasselman, T. Yap, K. Wathugala, G. Anderson, M.C. “A Top-Down Method for Uncertainty Quantification and Predictive Accuracy Assessment,” 46th Structures, Structural Dynamics, and Materials Conference Austin, TX April 18-21 2005
- Hazelrigg, G. A. “Thoughts on Model Validation for Engineering Design,” Proceedings DETC’03, ASME 2003 Design Engineering Technical Conferences and Computers and Information in Engineering Conference Chicago, Illinois September 2-6 2003
- Hills, R. G. Trucano, T. G. Statistical Validation of Engineering and Scientific Models: Background Sandia National Laboratories report, SAND99-1256 May 1999
- Hills, R.G. “Model Validation: Model Parameter and Measurement Uncertainty,” Journal of Heat Transfer April 2006 128 339 351
- Hodges, J. S. “Six (or so) Things You Can Do with a Bad Model,” Operations Research 39 3 May - June 1991
- Kennedy, M.C. O'Hagan, A. “Bayesian Calibration of Computer Models,” Journal of the Royal Statistical Society Series B - Statistical Methodology 63 3 2001 425 464
- Logan, R.W. Nitta, C.K. “Comparing 10 Methods for Solution Verification, and Linking to Model Validation,” Journal of Aerospace Computing, Information and Communication 3 2006 354 373
- Logan, R.W. Nitta, C.K. Chidester, S. K. “Uncertainty Quantification During Integral Validation: Estimating Parametric, Model Form, and Solution Contributions,” 47th AIAA/ASME/ASCE/AHS/ ASC Structures, Structural Dynamics, and Materials Conference May 1-4 2006 Newport, RI
- Mayes, R.L “Validation of a Blast Pressure Loading Model for a Shell-Payload Shock Response Model,” 2006 International Modal Analysis Conference (IMAC XXIV) Jan. 30 Feb. 2 2006 St. Louis, MO
- Oberkampf, W. L. Trucano, T. G. 2002 “Verification and Validation in Computational Fluid Dynamics,” Progress in Aerospace Sciences 38 3 209 272
- Oberkampf, W. L. Barone, M. F. 2004 “Measures of Agreement between Computation and Experiment: Validation Metrics,” AIAA 34th Fluid Dynamics Conference Portland, OR June 2004
- Roache, P.J. Verification and Validation in Computational Science and Engineering Hermosa Publishers Albuquerque, NM 1998
- Romero, V.J. “Characterization, Costing, and Selection of Uncertainty Propagation Methods for Use with Large Computational Physics Models,” presented at the 42nd Structures, Structural Dynamics, and Materials Conference Seattle, WA April 16-19 2001
- Romero, V.J. Sherman M.P. Dempsey J.F. Johnson J.D. Edwards L.R. Chen K.C. Baron R.V. King C.F. “Development and Validation of a Component Failure Model,” presented at the 46th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference April 18-21 2005 Austin, TX
- Romero, V.J. “On Model Validation and Extrapolation for Best-Estimate-Plus-Uncertainty Predictions,” Sandia National Laboratories document SAND2005-7678C November 2005
- Romero, V.J. “Some Issues in Quantification of Margins and Uncertainty (QMU) for Phenome-nologically Complex Coupled Systems,” presented at the 8th AIAA Non-Deterministic Approaches Conference May 1-4 2006 Newport, RI
- Romero, V.J. “Validated Model? Not So Fast. The Need for Model “Conditioning” as an Essential Addendum to Model Validation,” 9th AIAA Non-Deterministic Approaches Conference Honolulu, Hawaii April 23-26 2007
- Rutherford, B.M. Dowding K.J. “An Approach to Model Validation and Model-Based Prediction: Polyurethane Foam Case Study,” Sandia National Laboratories report SAND2003-2336 July 2003
- Tieszen, S.R. Domino S.P. Black A.R. “Validation of a Simple Turbulence Model Suitable for Closure of Temporally-Filtered Navier-Stokes Equations Using a Helium Plume,” Sandia National Laboratories report SAND2005-3210 June 2005
- Trucano, T.G. Pilch M. Oberkampf W.L. “General Concepts for Experimental Validation of ASCI Code Applications,” Sandia National Laboratories report SAND2002-0341 March 2002
- Trucano, T.G. Swiler L.P. Igusa T. Ober-kampf W.L. Pilch M. “Calibration, validation, and sensitivity analysis: What's what,” Reliability Engrng. and System Safety 91 11 1331 1357
- Urbina, A. Paez T.L. Smallwood D.O. “A Hierarchy of Validation Measures for Structural Dynamics,” 2006 International Modal Analysis Conference (IMAC XXIV) January 30 Feb. 2 2006 St. Louis, MO