Note: This essay is a synthesis of ideas and applications of Six Sigma, and QC Design. It assumes you are familiar with many of these concepts, but if you aren’t, there are links to essays and lessons with in-depth coverage. The main goal of this essay is to show how to put Method Validation together with Six Sigma Metrics in order to make a definitive decision on the suitability of a method for your own application.
There are dozens if not hundreds of factors to consider when purchasing of a new diagnostic instrument, and new issues are emerging all the time. As instruments get bigger and more expensive, they do more. Yet for all the marketing claims of faster, more advanced, more automated, more ”verified”, and integratedinternetworked, the critical characteristic of the device remains its analytical testing performance. (Remember – the device is supposed to perform a test accurately and precisely, not just produce test results faster and cheaper.) That performance can be judged through the manufacturer's precision claims.
However, what is the practical meaning of a precision claim? Is 2% good? How about 3%? Certainly if for test Z, instrument A has a precision claim of 2%, while instrument B has a claim of 3%, you have a better opinion of A. But is that a valid judgement? What is the real precision need for patient care?
The problem with precision claims is that while they can provide some relative comparison between competing products, they are disconnected from the real-world performance needs. By translating them into Six Sigma metric estimates, we can connect claims with the analytical needs of test performance.
When diagnostic manufacturers market a product, they are required to make a precision claim for each test. That, they must claim that the coefficient of variation (CV) of the device will be “X% or better.” It's one of the few quantitative claims for analytical performance they are required to make.
There are two ways to approach a precision claim: as an optomist or a pessimist. For the optimist, a precision claim is actually higher than the CV that the customer will routinely experience. Why? Because the claim has to be high to encompass all the different performances of the instrument across laboratories and locations. So precision claims are useful, but not worth worrying about.
For the pessimist, the precision claim must be taken at face value. If a manufacturer could claim a lower CV, they would, so there's a definite reason that the number is where and what it is. That is, if the manufacturer could claim a 1% CV, they would, so if they claim 2%, there's a reason for it. This is beyond worst-case scenario thinking – the claim indicates that the performance isn't significantly better in the majority of cases. Maybe some labs will be able to get the instrument to perform better, but many won't. So if you are getting a new instrument with a precision claim of 4% CV, you better plan that you will experience 4% CV when the device is installed in your lab.
Ultimately, a precision claim is only useful until you have better information. As soon as you can obtain performance data for the device, either through method validation studies or routine laboratory data, those numbers are far more useful than the precision claims. You want to know how the device will perform in your lab, not anyone else's.
A huge volume of work has accumulated about the topic of Six Sigma. There are several detailed articles about it in our public archives. To reduce (and oversimplify) Six Sigma, there now are "Sigma metrics" that provide a universal benchmark for process performance. The performance of all processes can be characterized on the "Sigma scale." Values typically range from 2 to 6, where the goal for "world class quality" is 6. If the Sigma metric is less than 3, you’ve essentially got a process that is so unreliable it shouldn’t be used for routine production. A process with a low sigma metric will cost you a lot of time and effort to maintain. To give you a benchmark for understanding process performance on the sigma scale, airline baggage handling is about a 4 sigma process. We hope that healthcare, including laboratory tests, has a better Sigma than that!
Let’s find out!
It's really quite simple. All it involves is division – and you only do it once. So if you remember what you learned in grade school, you can do this on a napkin.
Here’s the list of what you need:
From the manufacturer:
- Precision claim
From other sources:
For most cases, all you really need is the precision claim and the quality requirement. The manufacturer gives you the former. In a pinch, use your best judgement about the latter. But the good news is that you can find much of the needed data and tools right here on Westgard Web at no cost.
Let's start an example here. Pretend you are evaluating a new instrument that tests for Hemoglobin (Hb). The manufacturer's claim for precision is 1.5%. Is this good, bad, somewhere in the middle? Let's find out!
Again, this is a topic covered in a wealth of detail by other essays and lessons on this website. Quality requirement should be a self-explanatory term. However, in many US laboratories, the practice of defining the quality required by a test is rarely done, and the concept is almost unknown. No one denies that a logical quality process should begin by defining how "good" the process has to be. That is exactly what laboratories need to do. In some cases, the government has already taken this step. The CLIA rules for proficiency testing define acceptable limits for about 80 tests. These limits are in the form of an allowable total error that includes both the precision and accuracy of the test result. In other words, the effects of both the method CV and bias are included in the allowable total error.
Finding or defining quality requirements is a critical step in the QC Design or QC Planning Process. We refer you to those articles on the website for more explanation.
Continuing with our example, we find that for Hemoglobin, CLIA states that the results for should be good within +/- 7% of the true result.
[One important thing to note is that the CLIA quality requirements are most often given as percentages, which means the size of the error in concentration units actually gets larger as the concentration gets higher. Other times, the CLIA requirements specify an absolute value in concentration units, and sometimes, both are given, with the inference to use either the highest or lowest value appropriate at the concentration of interest. For example, with glucose, the CLIA requirement is stated as Target Value plus or minus 10% or 6 mg/dL, whichever is greater. That means the 10% figure would apply for any concentration above 60 mg/dL and the 6 mg/dL would apply to anything below 60 mg/dL. With Precision Claims, you aren't necessarily given the level at which that precision applies - so you have to choose which part of the CLIA requirement to use by deciding where the important medical decision level is.]
Now we’re ready to get Six Sigma metrics and will really be able to see how a test performs!
Here's where the division comes in. The gruesome details of how and why Six Sigma Metrics are related to bias, CV, and quality requirements are (yet again) covered by other essays on Westgard Web. Here we merely note that you can calculate Sigma metrics from performance data by this equation:
Sigma = (TEa – bias)/CV.
That’s pretty simple, isn’t it! It's even simpler for us, because we only have the precision claim at this point. Therefore, we're going to assume bias is zero. Is this a great idea to do? Not really, but when you've only got one number to go on, you do what you can. Hey, if you have a better idea or estimate of what bias is going to be, use it.
Enough discussion: let’s see the Sigma Metrics for our Hemoglobin example:
Sigma = (7-0)/1.5 = 4.6
4.6 is pretty good. 6 is ideal but usually not easy to achieve without a lot of effort - or reengineering.
As we stated earlier, this is just an estimate, and it's based on not only the manufacturer's claim but also on an assumption of zero bias.
The Sigma estimate at this point should be pretty good, because you'll need to make some “room”, or leave a safety margin, for an eventual bias.
If you have a low Sigma value at this point, that's not a good sign. If even the manufacturer's claims with zero bias don't allow you to meet a good Sigma metric, this means that you're going to have a very hard time achieving the necessary error detection on this test once you get it in your lab.
Sigma metrics calculated from a manufacturer's claims for performance may allow you to recognize poor performance and eliminate those methods from your consideration. The reliability of the metrics depends on whether the manufacturer's claims are truly representative of the performance that would be seen in your own laboratory.
In short, you need to get more information.
At the very least: try to get information on the bias claim.
Better: ask the manufacturer for a method validation study. That would give you “real” data on the performance of the instrument, including some CV and bias results. But keep in mind that the results of that method validation study are not from your lab, so you have no ability to .
Best: perform a method validation study in your lab.Then you'll have a very good idea of how well this instrument will perform. There is a way to convert method validation data into "real" Sigma metrics.
You can perform QC Design (we also call it QC Planning) using a tool like the Normalized OPSpecs charts available for free on the website, or the software programs QC Validator® 2.0, or EZ Rules®.
Since we only have a Six Sigma estimate at this point, any QC Design we perform at this stage will also be limited to an estimate. But given 1.5% CV and no bias, you could QC this method with 2 controls and limits set at 2.5s. This would provide at least 90% error detection, and 3% false rejection. See the OPSpecs chart below.
