METHOD VALIDATION - THE DECISION ON METHOD PERFORMANCE
James O. Westgard, Ph.D.
METHOD VALIDATION - You've performed the experiments, tabulated the results, plotted the data, and calculated the statistics. Now you have to make a decision on the acceptability of the method. How do you decide whether the method is good enough to use in your laboratory?
Remember the inner hidden deeper secret meaning of method validation - ERROR ASSESSMENT. The decision on the acceptability of method performance depends on the size of the observed errors relative to some "standard" or quality requirement that defines the medically allowable error. Method performance is acceptable when the observed errors are smaller than the medically allowable error. Method performance is NOT acceptable when the observed errors are larger than the medically allowable error.
You should actually define the medically allowable errors at important medical decision levels in the beginning to help guide the design of the experiments and the collection of the data. What will remain to be done, then, is to compare your observed errors with the defined medically allowable errors.
In the scientific literature, requirements for analytical quality have been defined in three different formats - allowable total error, allowable SD, and allowable bias. An allowable total error sets a limit on the combined effect of the random and systematic errors of a method, whereas an allowable SD and an allowable bias set separate limits for random and systematic errors, respectively.
Separate requirements for allowable SD and allowable bias would appear to be useful because these statistics can be calculated directly from the experimental data (e.g., an SD is calculated for the data from a replication experiment and a bias for the data from a comparison of methods experiment). However, the quality of a patient test result is determined by the net or total effect of both the random and systematic errors, therefore the total error is more relevant medically [1].
"The physician thinks rather in terms of the total analytical error, which includes both random and systematic components. From his point of view, all types of analytic error are acceptable as long as the total analytic error is less than a specified amount. Total error is medically more useful; after all, it makes little difference to the patient whether a laboratory value is in error because of random or systematic analytical error, and ultimately he is the one who must live with the error."
A common source is the external quality assessment survey or proficiency testing program in which you participate. These programs generally define a central "target value" and a range of values around that target that are considered acceptable. Because these programs usually ask for a single analysis on each survey specimen, both the random and systematic errors of your method will affect the results. The "acceptable range" is therefore an analytical performance requirement in the format of an allowable total error.
For US laboratories, the most readily available list of total error criteria are provided by the CLIA proficiency testing criteria for acceptable performance, which have been published in the Federal Register [2] and provide recommendations for some 80 different tests. See the list of criteria provided on this website. These criteria are presented in three different ways:
In a few cases, two sets of limits are given, e.g., the glucose requirement is target value plus or minus 6 mg/dL or plus or minus 10%, whichever is greater. At a medical decision level of 50 mg/dL, the allowable total error is 6 mg/dL or 12%. At a medical decision level of 125 mg/dL, the allowable total error is 10% or 12.5 mg/dL. For information on medical decision levels, see Dr. Statland's guidelines on this website.
To estimate the random error of the method from the replication experiment, you will have calculated an SD or CV. To estimate systematic error from the comparison of methods experiment, you will calculate the bias between the means obtained by the test and comparative methods, or will use regression statistics to calculate the expected difference at particular medical decision levels. These estimates of random and systematic errors need to be combined to judge their total effect.
The literature provides three different recommendations on how to combine random and systematic errors:
Rather than choose between these recommendations, all three can be utilized in a graphical decision tool, or a method decision chart [5]. The chart is simple to construct, minimizes the need for additional calculations, and provides a graphical picture that simplifies the interpretation and judgment on method performance.
First, express the allowable total error as a percentage of the medical decision concentration. Most CLIA allowable errors are already given in percent. For those given in concentration units, express the allowable error as a percent of the medical decision concentration of interest, i.e., divide the allowable error by the medical decision concentration and multiply by 100 to express as a percentage.
Next, take a sheet of graph paper
and do the following:
Express your observed SD and bias in percent, then plot the point whose x-coordinate is your observed imprecision and y-coordinate is your observed inaccuracy. This point is called the "operating point" because it describes how your method operates. You judge the performance of your method on the basis of the location of the operating point, as follows:
A. Albumin method with CV of 2.0%
and bias of 0.0% at 3.5 g/dL. The CLIA requirement for analytical
quality is 10%, therefore we can use a method decision chart
exactly like the one shown earlier. The operating point will
have a y-coordinate of 0.0% and an x-coordinate of 2.0%, as shown
by operating point "A". This method is clearly acceptable
and will be easy to control in routine operation.
Define the allowable total error for your test. Construct a method decision chart using a page of graph paper. Plot your observed inaccuracy (percent bias) from the comparison of methods experiment versus the observed imprecision (percent CV) from the replication experiment. See where this operating point is located and judge whether or not you want to implement the method for routine service.
You can also use the Normalized Operating Point Calculator on this website to calculate your observed inaccuracy and observed imprecision as a percentage of the allowable total error, print the accompanying Normalized Method Decision Chart, and then plot your normalized operating point on that normalized chart. Again, see where this operating point is located and judge whether or not you want to implement the method for routine service. The outcome will be the same as from a regular (non-normalized) decision chart.
