In order to determine the contribution of a process improvement or new technology to a military practice, product, or mission area it is critical to define and assess rigorous measures of merit. Only through using such structured analysis is it possible to properly evaluate mission contribution or cost effectiveness. Most efforts to assess such contribution have either used very narrowly applicable numeric indices or generally stated anecdotal evidence.

Developing well defined and balanced metrics that reflect changes across the relevant spectrum of warfighting capabilities and skills is required. Only by employing such comprehensive and rigorous measures can the impact of system and process changes be accurately assessed. Such assessments are essential as US force structure and employment concepts adapt to future challenges. [top}

Technical Challenge

Currently most assessments use metrics that are uneven in scope, very case specific, or not well structured. They are do not reflect the value of a change in all of the areas impacted nor do they allow aggregation or generalization. Additionally many measures that are commonly used, like return on investment, are actually misused and are others are undefined, thus making the assertions of value at best vague and at worst incorrect.

Therefore, the use of a thorough analytic approach to assess the real contribution of a system in support of military missions is absolutely critical. Such an analysis will allow questions of a system’s contribution to be answered in a rigorous, defendable, and comprehensive manner across Service and Department of Defense applications. [top}

Military Application

Given the importance of measuring system effectiveness, there is a pressing need to carefully understand its characteristics and constraints, to gather new data, and to place such data into a construct that allows defensible assessments of military readiness. In the context of the military, readiness is the ability of a unit or group of units to perform its (their) assigned mission areas and assessments of it consider affordability, risk, and reward.

Readiness has, traditionally, been assigned a numeric value for commanders to quickly assess force status and their ability to meet the challenges of required national tasking. Various functional areas are measured such as training, force levels (personnel), equipment status, and mobility. Such sub-areas are then aggregated into a single measure of overall readiness. These measures have been a mix of subjective and objective determinations that, at times, do not truly reflect unit readiness nor adequately, especially in the case of training, measure the quality of the readiness level.

The need is for a more rigorous and objective system of metrics that reflects demonstrable changes in readiness over time through the use of an additive, hierarchical, approach. To accomplish this requires analysis that compares the current state of readiness and its maintenance with one that is enhanced by a new process, technique or technology. The key difference here is “measurable” which implies metrics. The output, that is enhanced readiness, must be readily measured by agreed upon (and hopefully objective) metrics. Then, it is incumbent upon commanders to decide which skills and warfighting capabilities are most relevant and to train and support those areas to the highest level—perhaps at the expense of others. The result of applying such structured assessment is an improved ability to measure of the both the quantity and quality of the readiness of operational forces. [top}

Technical Approach

The approach VisiTech uses is to develop a rigorous structure of metrics and then, through case study examinations, develop individual values and application specific aggregations of system contribution. In this way it is possible to have both the specific details needed for defending numeric indicators of value while also generating metrics that reflect the application of a system to a more general mission—like team training.

To develop metrics that reflect a system’s contribution requires a thorough understanding of the available evidence, assessment dimensions, and the military context. Evidence available include qualitative and quantitative indices; the domains of assessment consist of both the process being conducted and the resulting outcomes; and finally, the military context reflects mission readiness and proficiency.

Structured assessment also requires the ability to make important distinctions between confusing terms like metrics, measurement, measure of merit, metric type, and measurement hierarchy (to name a few). When these distinctions are made clear with defined terms and well-documented case studies within the context of an overall structure of value assessment, the conclusions are convincing.

Two key terms in assessing a system’s contribution are metrics and measurement. Metrics are a standard of measurement (e.g., inches or yards for distance). Measurement is the dimension or quantity of that which is being assessed. Thus, we need to construct the standards (defining the metrics), empirically measure the relevant processes and output and then display and often aggregate the results into indexes that are meaningful to the warfighter. The value of a system can then be demonstrated.

To specify the metrics needed to assess a system capability requires four components: measures, measure structure, measure association, and measure assessment—of both output and process. Each needs to be very carefully defined. We accomplish this through use of carefully tailored derivations of adopted operations research terms and references, expanding and linking them in innovative and unique ways, and assessing them in both controlled and real world environments. The linking of the measures is especially important in assessing the contribution and thus requires very careful association to the mission being supported.

In assessing the effectiveness of military activities, VisiTech, Ltd., has significant expertise and experience in metric development and assessment. We can help you to define the assessment dimensions, collect the data, and analyze the results. [top}

Project Experience

In 1998, then RADM Nutwell, Deputy Director of the Navy’s Space, Information Warfare Command and Control Directorate (N6B) asked a fundamental yet difficult question: “What is a pound of C4ISR worth?” As a result, a Military Operations Research Society (MORS) special session on C4ISR was planned, conducted, and conclusions reached. VisiTech, Ltd., personnel were an integral part of that event, from meeting with RADM Nutwell to refine his requirements, through developing the needed Terms of Reference, attending the conference, and drawing conclusions and insights. VisiTech developed measure of merit term definitions, hierarchical structures, sample metrics within both C4ISR and warfighting, and also distilled the results of efforts to link the two. That is, associate, through well-defined metrics, the warfighting impact of changes in C4ISR capabilities. As a result of this effort, C4ISR metrics are better defined, understood, and linked to warfighting.

Also, since 1999, VisiTech has supported NAVMSMO’s efforts to quantitatively assess the contribution of M&S to the successful conduct of Navy missions. This effort has resulted in a well-structured analysis framework and case study products. The analysis framework divides M&S metrics into anecdotal, economic, and comprehensive types. For each, definitions and decompositions of applicable measurement types are developed. Then selected measurement data are gathered from operational M&S applications. Finally, equations are proposed to allow the combination of multiple metrics into an aggregate measure.

Currently, VisiTech is supporting a Commander, Fleet Forces Command (CFFC) effort to develop and assess metrics that reflect the contribution of M&S to meeting Navy Intermediate Phase Training goals. The approaches being developed is to baseline the current metric types and data gathered historically, modify and extend, and then apply the new set to the Battle Group In-Port Exercise events now being included in this phase of the inter-deployment training cycle. These new metrics take into account the use of the Battle Force Tactical Trainer (BFTT) and similar distributed and on-board simulation systems.

In another ongoing effort, VisiTech is developing for the Defense Modeling and Simulation Management Office a set of descriptors and numeric parameters that characterize a simulation’s algorithms and characteristics. In this way, the key parameters for interoperability and application reuse are defined and described. [top}