Quality Standards

The debate on performance specification (quality goal) models has been going for decades. What's the bottom line? If we used permissible uncertainty instead of allowable total error, would our judgments on method accepability be any different? 

Comparing Verdicts from Different Goals, Models, and Specifications - Roche cobas 8000 c701

February 2016
Sten Westgard, MS

As we debate the best way to determine ideal and practical performance specifications (quality goals or quality requirements), it may be useful to come down out of the clouds and look at some actual laboratory examples. In a previous article, we analyzed the Roche cobas 8000 c701 using Sigma-metrics and allowable total error performance specifications (mainly from the US CLIA program). Now we're going to look at the same data, but use different mdoels and specifications. Will the different models mean a different verdict? Let's see...

The original paper can be found here:

Importance of implementing an analytical quality control system in a core laboratory.Marques-Garcia F, Garcia-Codesal F, del Rosario Caro-Narros M, Contreras-SanFeliciano T,  Revista de Caldidad Asistencial 2015 Nov-Dec;30(6):302-9.

The Sigma-metric analysis can be found here. https://www.westgard.com/cobas-c701.htm

Dr. Rainer Haeckel and Werner Wosniok et al published a set of permissible limits for measurement uncertainty last year (2015). This includes permissible imprecision (pCV%), permissible bias (pB%), and permissable standard and expanded measurement uncertainty (pU%). In addition to that article, the authors have a spreadsheet that provides a more comphrensive set of specifications than their journal article. Note that these target measurement uncertainties are supposed to be based on the level of interest within the reference range. Haeckel, Wosniok et al provide these examples based on some textbook reference ranges and their own example control levels.

We'll calculate measurement uncertainty simply as the square root of the sum of the square of the observed imprecision and square of the observed bias. This is an unorthodox calculation of measurement uncertainty, since the usual prescription is to eliminate any bias. In this case, we're going to incorporate the bias into the measurement uncertainty calculation as if was any other uncertainty.

*indicates that these Sigma-metrics were generated by using Ricos goals or CAP goals - for those analytes a CLIA goal was not available.

Anything highlight in red exceeds an allowable performance specification.

Now, we can further evaluate this same performance by making simple comparisons of the imprecision and bias against the desirable specifications for imprecision and bias (Ricos goals).

 Notice that just by using the simple desirable specifications for the individual components of error, we have determined that more assays are unacceptable than by measurement uncertainty terms or by Sigma-metrics.

Here's an overview of the assays and which ones are unacceptable according to each goal source.

The measurement uncertainty goals are the most stringent, followed by the Ricos imprecision and bias specifications, with Sigma-metrics coming in last. But while overall the number of methods considered unacceptable is nearly the same (only 1 or 2 methods different), which methods are considered unacceptable are different depending on the goals being used.

CLIA goals are often derided as being too wide and lenient, but clearly using the Sigma-metric calculation is still a demanding benchmark. Individual error parameters by Ricos goals are slightly more demanding. The target measurement uncertainty performance specifications are even more challenging still. It begs the question: if one of the latest instruments on the market has between a third and nearly a half of 23 biochemistry analytes considered unacceptable, does it matter which set of goals we use? We're not going to turn off half of our testing methods - we may need to accept that our goals are too tight, no matter what the model, no matter how we calculate performance, and scale back our expectations, and, most importantly, those of our clinicians. We may need to re-educate clinicians on how they interpret these test results, because they may currently be making a lot of decisions based on noise, not signal.