VerificationAndValidation

While incredible useful and powerful, numerical tools such as OpenSees can produce misleading and wrong results. This can be due to errors in the application code, the unsuitability of the users modeling choices for the system being analyzed, or user error in the input file. What this document provides are example code that have been used to verify the correctness of the code, i.e. it's verification. Examples in the literature exist that demonstrate the validity of the modules for a limited number of different systems. Without taking control of the analysis away from you, there is little we can do to protect you from yourself (inappropriate choice of model and user error!)

Before we begin, here are what we mean by Verification and Validation:

1. Verification of a software component is the process of comparing the results for that component with a known analytical solution. Our definition of verification, sometimes called formal verification is in agreement with the Wikipedia definition: "is the act of proving or disproving the correctness of intended algorithms underlying a system with respect to a certain formal specification or property, using formal methods of mathematics"
2. Validation of the model deals with comparing how well the model compares with results for the actual physical system that the model is being used to predict the behaviour of. This is in agreement with the American Association of Mechanical Engineers (ASME) definition of the validation process: “Validation can be defined as the process of determining the degree to which a model is an accurate representation of the real word from the perspective of the intended uses of the model”. Two things to note: (1) the model can be validated, not the software which takes as input the model, and (2) that the assessment of the accuracy between the simulation and the real world is subjective and has to be made relatively to the "intended use" of that particular model.

As you see, both verification and validation are related to a comparison process. They differ from each other for the type of results against which the comparison is carried out. Verification consists in comparing the numerical results against a known theoretical solution of the specific problem under investigation, while the validation refers to the comparison of the numerical model with a physical experiment which reproduces the particular phenomenon under investigation. As such, verification is not also a condition sufficient to guarantee that results obtained from the numerical model are correct. The theoretical model may itself be incorrect!, the assumptions it makes may not be suited to your system or may not be suited to the response quantities of interest to you. Hence, once a model has been verified, it is also necessary to validate the results against one or more experimental tests. In our field, given the limited number of experimental tests and the fact that no test probably exists for the exact system being modeled, the validation that has been performed is typically limited to different systems than your own. Great care must then be taken by you the user in determining the accuracy of the results for your particular model. (Between you and me, and given what you know about validation, always treat with suspicion anyone would would state that there application code has been fully validated!)

The Verification of software is an important topic, it is also a time consuming endeavour and we do the best we can with the limited resources available. For OpenSees we typically do dynamic black-box testing of the code, in which we execute the OpenSees application with a given set of test cases that endeavour to cover as much of the possible code-branches as possible. In should be noted, that even if all code branches are tested it is still not a 100% gaurantee of correctness as the same code block may produce results correctly or incorrectly depending on the variable values (again be careful when someone states their code has been fully verified!). In addition some white-box unit and systemtesting" has been performed. These are the examples written in C++ and found in the source code repository.

This following presents the set of simple systems that have been analyzed using OpenSees for verification purposes. While few in number, as you will see if you look at the reference documents, they are probably more exhaustive than some of commercial applications! The OpenSees results are compared against the results from other sources (verification documents for other finite element and structural analysis applications, examples in the literature, and hand calculations). The script examples cover a number of element and material types, solvers, integrators, solution algorithms, static, dynamic and eigenvalue analysis. For each system a number of verification scripts may be provided or multiple analysis may be perfomed in a single script (e.g. one script could perform repeated analysis using different solvers for each analysis to verify the workings of the different solvers).