U.S. Department of Energy

Pacific Northwest National Laboratory

Signature Quality Assessment

Assessing the quality of a signature is key to determining whether the signature is accomplishing its goal or whether the classifier needs to be revised to better identify, characterize, or detect the signature.

A formal assessment of signature quality involves constructing a multi-attribute utility function to compare two or more signature systems. The utility function accounts for all the measurable criteria, or attributes, that will be used to evaluate the systems. The multi-attribute utility function is the weighted average of the single attribute utility functions.

A variety of methods can be used to construct a multi-attribute utility function.  A traditional approach is to: 

  1. Identify and measure attributes of the quality of the signature system.  Example attributes include:
    • Fidelity:  How well does the classifier predict, detect, or characterize the phenomenon of interest?
    • Cost:  How much does it cost to make a prediction for a single observational unit?
    • Risk:  What are the likelihood and consequences of misclassification?
    • Other Attributes:  Are there other measurable attributes or criteria that would be used to distinguish one signature system from another, especially in an operational environment?  Examples include weight of the sensor, battery life, risk to personnel when using the system, and the amount of a forensic sample consumed during measurement.
  2. Construct single-attribute utility functions that map each attribute to the [0, 1] interval, such that 1 is highest utility and 0 is lowest utility.  The curvature of the single attribute utility function is chosen to reflect the relative value of the attribute as it moves through its observable range.
  3. Identify relative weights for each attribute that reflect the tradeoffs among these attributes. For example, the weights must reflect our preferences for a signature system that performs well with respect to one attribute (e.g., fidelity) at the expense another attribute (e.g., cost).
| Pacific Northwest National Laboratory