Quality Requirements and Standards

Here's a recent example proving the importance of the choice of quality requirement. A laboratory was reviewing the performance of a cholesterol method. By one local standard, the performance was not acceptable. By another standard (CLIA), it was. So is the method good or not? Which quality requirements are right? This case study looks at a number of quality requirements and compares them.

What's the Right Quality Goal? An Example

Sten Westgard, MS
September 2012

• What is the quality required by a cholesterol method?
• What is the performance of a real-world cholesterol method?
• What is the Sigma-metric?
• What QC is necessary?
• Which goal is the right goal?

Watching the 2012 summer Olympics, it is remarkable to see so many athletes from so many places around the world, come together and compete. In a recent New Yorker article, Louis Menand noted that this is due to an important topic familiar to laboratory professionals: standardizarion.

"[O]ne of the ways games get turned into sports is standardization. A sport is a game that, no matter where it takes place, is always played by the same rules... Regulation and standardization insure that, when competitors meet, everyone will have trained for the same event, and the results can be compared across competitions. Standardization is what makes it possible to have world records." [Glory Days, Louis Menand, The New Yorker, August 6, 2012, pages 64 - 72]

So the Olympics have one of the key concepts that laboratories lack. Every gold, silver, and bronze medal is "traceable" to a standard established for that sport. Put differently, we may never get to declare a laboratory worthy of a gold medal until we've established good standardization.

This lack of standardization is becoming a more important issue when it comes to quality requirements. In an essay last year, we discussed the modern conundrum about quality requirements: what's the right goal for a method? Laboratories are blessed now with many, many sources of quality requirements: the CLIA proficiency testing criteria, the German Rilibak, the biological variation database (sometimes called the 'Ricos Goals'), which further defines specifications for minimum, desirable, and optimal performance. This year we also saw the publication of a minimum set of quality specifications from the same Spanish group that maintains the biologically-derived quality specifications. This minimum set isn't based on biological variation; instead, it's based on an analysis of historical data that would allow 90% of laboratories to pass EQA.

What's the quality required for a cholesterol method?

To give a more concrete example of this issue, let's take a common lab test, cholesterol, and look at some of the different quality requirements available for this test.

As you can see, there is a wide variation of goals. The smallest goal is less than half, almost a third, of the largest goal.

Those difference might not seem that large or significant. But what happens when we bring Sigma-metrics into the picture? That calculation is driven by the quality requirement as well as observed imprecision and bias. So if the quality requirement changes, the Sigma-metric will change.

What's the performance of a real-world cholesterol method?

We were recently fortunate to collaborate with a laboratory in Canada that agreed to share some of their test data. So let's use the data on their cholesterol method from their high-volume analyzer. Routine controls are run on this method at two levels - 6.84 mmol/L and 2.74 mmol/L. Routine control data collected over approximately 3 months give us good estimates of imprecision of the method. The method is also part of a CAP proficiency testing (PT) survey, and relevant samples measured near those two levels of interest were used to give an estimate of bias. This performance data is detailed in the following summary:

Now we have a good set of numbers. We have performance measurements for imprecision and bias and quality requirements. This is everything we need to calculate Sigma-metrics.

What's the Sigma-metric of this cholesterol method?

Recall that the Sigma-metric equation is  (TEa - bias ) / imprecision

Let's take the performance data at the high control level and use the Rilibak quality requirement of 13%

Thus, for the high level, Sigma is  (13- 0.08) / 1.15

= 12.92 / 1.15

= 11.23 Sigma

Thus, using the Rilibak quality requirements, we would judge this method very favorably. We would characterize this method as better than world class.

However, obviously the different quality requirements will end up generating different Sigma-metrics. Here's the summary:

We can see that, if the quality requirements from Rilibak, the Spanish consensus minimums, CLIA or even the desirable specification from the biologic variation database, were applied, we would give high marks to this method. If we applied RCPA goals or QMP-LS goals, we might find performance at the high end to be acceptable, possibly even good, but on the low end we would not be satisfied with this method.

If you want to see this visually, we can use a Normalized Method Decision chart, which shows the performance data (remember this is the exact same performance data each time) put into the context of the quality requirement. Here's the chart:

In this graph, the dots shown have colors that correspond to different the quality requirements as listed on the right side of the graph.

The two "strings" of data correspond to the high level (the lowest string of points) and the low level (the higher string of data).

What QC is necessary for a cholesterol method?

In addition to impacting the judgment of the method, the quality requirement also helps determine what QC is necessary. For different quality requirements, different QC procedures will be needed to achieve the level of quality assurance.

Using an OPSpecs chart, normalized as above, we can make a similar display of the different quality requirements:

As you can see, the prescription for QC also varies with the quality requirement. Here we used the performance at the low level to design QC. Performance at the high level is universally good (at least), but performance at the low level is where there is differentiation, which would need different amounts of QC effort.

If we use the tightest quality requirement, our QC procedures won't be able to achieve the desired level of assurance; defects that exceed that allowable error won't be detected immediately. Those errors would grow and persist over several runs before we are able to catch them. If, on the other hand, we use a more "tolerant" quality requirement, we will be able to use wider control limits that have low false rejection rates as well as high error detection. When a method exceeds the allowable error, we'll be able to catch that error within the first run that it occurs.

Which is the right goal?

The 1999 Stockholm consensus heirarchy helped to rank the quality requirements in order of preference. The goals based on biologic variation are judged to be better than those based on "state of the art" and consensus proficiency testing goals. So the Desirable goal is a good choice of the quality requirements listed here. In this case, it also happens to be conveniently in about the middle of the range of available quality requirements.

There is also the possibility that some quality requirements are not appropriate for Sigma-metrics. When some PT or EQA groups set allowable errors, they don't have in mind that the laboratory should achieve performance at the level of Six Sigma. Instead, they may be aiming for a lower Sigma level such as 95% or 90% achievement (which would be 1.7 and 1.3 Sigma on the long-term scale, respectively, or 3.1 and 2.8 Sigma on the short-term scale, respectively) So while some organizations are setting tighter goals, they may be allowing more misses. Which raises a practical question for the laboratory: What goal is preferable? A larger (wider) goal that the laboratory can hit much more consistently? Or a smaller goal that the laboratory can hit less frequently? Just how small do the goals need to be and what level of success at hitting them is necessary?

Given the wide variety of medical practice around the world, there's no easy answer. Labs that follow strict glycemic control protocols are going to set different quality requirements than labs that use glucose measurements less stringently. There is no monolithic organization that commands the creation and use of quality requirements. In many ways, that's good, because it allows individual laboratories significant freedom. But it does mean that for our present, using Sigma-metrics means also paying close attention to what quality requirements were chosen and why those goals were chosen. It's easy to get a 6 Sigma method if you simply search out the widest possible quality goal and hide that choice in a footnote.

As we noted in the earlier essay on choosing the right quality goals, it is tempting to let perfection be the enemy of the good. Because there is so much variety in the quality goals, laboratories may feel an urge to throw up their hands and avoid using any quality requirement. If we translate this situation to another realm, this approach is a bit dubious. For instance, if we found that we couldn't agree on the best choice of speed limit for our highways, does that mean we should avoid setting any speed limit (i.e. the Montana approach). Simply because making a choice is not easy, does that mean we should avoid making any choice at all?

Without doubt, quality goals in laboratory medicine need to evolve, improve, and eventually standardize. This is true about many different aspects of the laboratory. That means our current use of quality goals requires an extra step - we have to make sure we choose the right goal.