The goal of project is to produce a system that supplements existing verification and validation (V&V) techniques by improving the characterization of models and simulations. There are several situations in which the current V&V techniques provide characterizations that are less than wholly satisfactory. The major situations for which current V&V techniques are lacking are:

This project has developed an automated method of evaluating the performance of simulations under various operating conditions. For many of today’s simulations, runs under identical or nearly identical initial and operational conditions can result in significant differences in the quality of the representation provided by the model. This variability could come from such sources as the stochastic nature of the simulation itself or the variability inherent in passing information around a network. The result is that these simulations are stochastic in nature. Therefore, the system developed under this effort defines the simulation’s representation of reality in a probabilistic manner. That is, it determines appropriate probability functions that characterize the simulation.

A simulation’s representation of reality is dependent upon the conditions under which it was run. That is, there are some key factors that affect the performance of the simulation. The probability that the simulation represents any specific aspect of interest accurately is dependent upon these key factors. Therefore, in defining the probability density functions that characterize the simulation, the key performance factors are treated as independent variables or the conditions of conditional probability functions. The aspects of interest to be observed or measured from the simulation are treated as dependent variables. That is, the probability of their valid representation is dependent on the key performance factors.

Different aspects of interest to be observed or measured from the simulation are likely to have different dependencies on the key performance factors. That is, it is possible that under a given set of operational conditions, one aspect of interest may have a high probability of valid representation while such a probability for another aspect may be low. For this reason, separate probability functions are be generated for each identified aspect of interest.

Being statistically based, many simulation runs are needed to generate the characterization. The system developed automates the characterization process to minimize the staff hours required. This automation includes simulation initialization and control, data collection, and data analysis. To minimize the time needed for characterization while also ensuring it is complete, heuristic search routines control the modification of the operating conditions throughout the solution space.

The characterizations of the simulation produced by this system can be used for:

	Employment guidance; matching scenario size with hardware or bandwidth resources available
	Systems engineering; estimate effects of architecture or hardware changes
	Interoperability; identification of federation agreement compliance and establishing federation use guidance for scenarios, topology, etc.
	Exercise quality assurance during runtime
	Post runtime assessment of the quality of the simulation with respect to its value of training or potential to support analyses