This content is not included in
your SAE MOBILUS subscription, or you are not logged in.
Type X and Y Errors and Data & Model Conditioning for Systematic Uncertainty in Model Calibration, Validation, and Extrapolation
1
Technical Paper
2008-01-1368
ISSN: 0148-7191, e-ISSN: 2688-3627
Annotation ability available
Sector:
Language:
English
Abstract
This paper introduces and develops the concept of “Type X” and “Type Y” errors in model validation and calibration, and their implications on extrapolative prediction. Type X error is non-detection of model bias because it is effectively hidden by the uncertainty in the experiments. Possible deleterious effects of Type X error can be avoided by mapping uncertainty into the model until it envelopes the potential model bias, but this likely assigns a larger uncertainty than is needed to account for the actual bias (Type Y error). A philosophy of Best Estimate + Uncertainty modeling and prediction is probably best supported by taking the conservative choice of guarding against Type X error while accepting the downside of incurring Type Y error. An associated methodology involving data- and model- conditioning is presented and tested on a simple but rich test problem. The methodology is shown to appropriately contend with model bias under conditions of systematic experimental input uncertainty in the test problem. The methodology effectively bounds the uncertain model bias and brings a correction into the model that extrapolates very well under a large variety of extrapolation conditions. The methodology has been straightforwardly applied to considerably more complex real problems where system response is likewise jointly monotonic in the input uncertainties. The methodology also allows for other types of systematic and random uncertainty in the experiments and model as discussed herein.
Citation
Romero, V., "Type X and Y Errors and Data & Model Conditioning for Systematic Uncertainty in Model Calibration, Validation, and ExtrapolationAlso In
Reliability and Robust Design in Automotive Engineering, 2008
Number: SP-2170; Published: 2008-04-14
Number: SP-2170; Published: 2008-04-14
References
- AIAA 1998 Guide for the Verification and Validation of Computational Fluid Dynamics Simulations AIAA G-077-1998, American Institute of Aeronautics and Astonautics Reston, Va.
- ASME V&V 10 - 2006 Guide for Verification and Validation in Computational Solid Mechanics http://catalog.asme.org/Codes/PrintBook/VV_10_2006_Guide_Verification.cfm
- Coleman, H.W. Stern, F. “Uncertainties in CFD Code Validation,” Journal of Fluids Engineering Dec. 1997 119 795 803
- Department of Defense 2006 Key Concepts of Verification, Verification, & Accreditation VV&A RPG Sept. 15 2006
- Easterling, R. G. “Measuring the Predictive Capability of Computational Models: Principles and Methods, Issues and Illustrations,” Sandia National Laboratores report, SAND2001-0243, Unlimited Release February 2001
- Hasselman, T. Yap, K. Wathugala, G. Anderson, M.C. “A Top-Down Method for Uncertainty Quantification and Predictive Accuracy Assessment,” paper AIAA-2005-1903, 46th Structures, Structural Dynamics, and Materials Conference Austin, TX April 18-21 2005
- Hazelrigg, G. A. “Thoughts on Model Validation for Engineering Design,” Proceedings DETC'03, ASME 2003 Design Engineering Technical Conferences and Computers and Information in Engineering Conference Chicago, Illinois September 2-6 2003
- Hills, R.G. Dowding, K.J. “Statistical Validation of Engineering and Scientific Models: Bounds, Calibration, and Extrapolation,” Sandia National Laboratories document SAND2005-1826 April 2005
- Logan, R.W. Nitta, C.K. Chidester, S. K. “Uncertainty Quantification During Integral Validation: Estimating Parametric, Model Form, and Solution Contributions,” 47th AIAA/ASME/ASCE/AHS/ ASC Structures, Structural Dynamics, and Materials Conference May 1-4 2006 Newport, RI
- Oberkampf, W. L. Trucano, T. G. 2002 “Verification and Validation in Computational Fluid Dynamics,” Progress in Aerospace Sciences 38 3 209 272
- Roache, P.J. Verification and Validation in Computational Science and Engineering Hermosa Publishers Albuquerque, NM 1998
- Romero, V.J. “On Model Validation and Extrapolation for Best-Estimate-Plus-Uncertainty Predictions,” Sandia National Laboratories document SAND2005-7678C November 2005
- Romero, V.J. “A Paradigm of Model Validation and Validated Models for Best-Estimate-Plus-Uncertainty Predictions in Systems Engineering,” Paper 200-01-1746 for Soc. Automotive Engineers 2007 World Congress April 16-20 Detroit, Michigan
- Romero, V.J. “Validated Model? Not So Fast. The Need for Model ‘Conditioning’ as an Essential Addendum to Model Validation,” paper AIAA-2007-1953, 9th Non-Deterministic Approaches Conference Honolulu, HI April 23-26 2007
- Romero, V.J. “A Paradigm of Real-Space Model Validation and Model Conditioning for Best-Estimate-Plus-Uncertainty Predictions in Hierarchical Modeling,” Computer Methods in Applied Mechanics and Engineering
- Rutherford, B.M. Dowding K.J. “An Approach to Model Validation and Model-Based Prediction: Polyurethane Foam Case Study,” Sandia National Laboratories report SAND2003-2336 July 2003
- Trucano, T.G. Swiler L.P. Igusa T. Oberkampf W.L. Pilch M. “Calibration, validation, and sensitivity analysis: What's what,” Reliability Engrng. and System Safety 91 11 1331 1357
- Urbina, A. Paez T.L. Smallwood D.O. “A Hierarchy of Validation Measures for Structural Dynamics,” paper #181 of the 2006 International Modal Analysis Conference (IMAC XXIV) January 30 Feb. 2 2006 St. Louis, MO
- Byars, E.F. Snyder, R.D. Plants, H.L. 1983 Engineering Mechanics of Deformable Bodies Harper & Row Publishers NY
- Ferson, S. Kreinovich V. Hajagos J. Oberkampf W. Ginzburg L. “Experimental Uncertainty Estimation and Statistics for Data Having Interval Uncertainty,” Sandia National Laboratories report SAND2007-0939 May 2007
- Engineering Statistics Handbook National Institute of Standards and Technology web site
- Blottner, F.G. Lopez, A.R. “Determination of Solution Accuracy of Numerical Schemes as Part of Code and Calculation Verification,” Sandia National Laboratories report SAND98-2222 Oct. 1998
- Roy, C.J. “Review of Code and Solution Verification Procedures for Computational Simulation,” J. Computational Physics 205 2005 131 156
- Salas, M.D. “Some Observations on Grid Convergence,” Computers & Fluids 35 2006 688 692
- Romero, V.J. “Efficient Global Optimization Under Conditions of Noise and Uncertainty - A Multi-Model, Multi-Grid Windowing Approach,” Proceedings of the 3rd WCSMO (World Congress of Structural and Multidisciplinary Optimization) Conference Buffalo, NY May 17-21 1999
- Romero, V.J. “Characterization, Costing, and Selection of Uncertainty Propagation Methods for Use with Large Computational Physics Models,” paper AIAA-2001-1653, 42nd Structures, Structural Dynamics, and Materials Conference Seattle, WA April 16-19 2001
- Romero, V.J. “Model-Discretization Sizing and Calculation Verification for Multipoint Simulations over Large Parameter Spaces,” paper AIAA2007-1937, 9 th Non-Deterministic Approaches Conference Honolulu, HI April 23-26 2007
- Logan, R.W. Nitta, C.K. “Solution Verification Linked to Model Validation, Reliability, and Confidence,” paper 2005-01-1774 of Soc. Automotive Engineers 2005 World Congress April 11-15 Detroit, MI
- Maupin, R. “Calculation Verification for Threaded Assembly,” IMAC-XXIV Conference on Structural Dynamics Jan. 30 Feb. 2 2006 St. Louis, MO
- Roy, C.J. Raju, A. Hopkins, M.H. “Estimation of Discretization Errors Using the Method of Nearby Problems,” AIAA Journal 45 6 June 2007
- Stern, F. Coleman, H.W. Wilson, R.V. Paterson, E.G. “Verification and Validation of CFD Simulations,” Proceedings of the 3 rd ASME/JSME Joint Fluids Engineering Conference July 18-23 1999 San Francisco, CA
- Stewart, J.R. Gullerud, A.S. Heinstein, M.W. “Solution Verification for Explicit Transient Dynamics Problems in the presence of Hourglass and Contact Forces,” Comput. Mthds. Appl. Mech. Engrg. 195 2006 1499 1516
- Campbell, K. “A Brief Survey of Statistical Model Calibration Ideas,” International Conference on Sensitivity Analysis of Model Output March 8 - 11 2004 Santa Fe, NM
- Romero, V.J. “Type X and Y Errors and Data & Model Conditioning for Multivariate Systematic Uncertainty in Model Calibration, Validation, and Extrapolation,”