Sigma Metric Analysis
Sigma Estimates: High Volume Hematology
Sten Westgard, MS
Working with CAP Today's recent survey of High Volume Hematology Analyzers, we convert the manufacturer precision claims into Six Sigma estimates. Want to see how the instruments shape up?
FROM PRECISION CLAIMS TO SIX SIGMA ESTIMATES: HIGH VOLUME HEMATOLOGY ANALYZERS
- What data was used and how was it collected?
- Results
- What are the possible consequences of these Sigma metric estimates?
- Observations, conclustions, and important notes
- Recap: What do you need to go from Manufacturer Claims to Sigma metrics?
- Want to learn more about Six Sigma?
March 2004, Updated January 2005. Updated January 2006.
Some error detection probabilities corrected September 2005
 |
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.] |  |
CAP Today has one of the more comprehensive market surveys available in the healthcare industry. Occasionally, the journal also collects analytical data such as precision claims. Since we've recently posted an article on how to convert precision claims into Six Sigma estimates, we thought it would be useful to put that lesson into action, using the public data available in the CAP Today surveys. The focus of this QC application will be on high volume hematology analyzers, covered in the 2004 issue of CAP Today. .
For the purposes of this application, we are not going to walk through the steps of the calculations (see the previous article links or the recap below to see how these numbers are calculated). We will only present the final results here and discuss them.
What data was used and how was it collected?
The CAP Today survey collects data provided by the manufacturers themselves. These companies have given their precision claims. In fact, manufacturers are required to provide precision claims when they market a diagnostic device.
Note that in the CAP Today survey, no details of how these precision claims were created are available. We, as readers, must accept them on faith.
Results
Here are the Precision Claims along with the Sigma metrics:
| Analyte | RBC | WBC | Hb | Pt | ||||
| Erthrocytes | leukocytes | Hemoglobin | Platelet Count | |||||
| Total Allowable Error % | +/- 6% | +/- 15% | +/- 7% | +/- 25% | ||||
| Instrument | Precision Claim | Sigma Estimate | Precision Claim | Sigma Estimate | Precision Claim | Sigma Estimate | Precision Claim | Sigma Estimate |
| Abbott | ||||||||
| Cell-Dyn Ruby | 1.80 | 3.33 | 2.40 | 6.25 | 1.40 | 5.00 | 3.80 | 6.58 |
| Cell-Dyn Sapphire | 1.50 | 4.00 | 2.70 | 5.50 | 1.00 | 7.00 | 4.00 | 6.25 |
| Cell-Dyn 3200 | 1.50 | 4.00 | 2.70 | 5.50 | 1.00 | 7.00 | 4.00 | 6.25 |
| Cell-Dyn 3700 | 1.50 | 4.00 | 2.50 | 6.00 | 1.20 | 5.80 | 5.00 | 5.00 |
| Cell-Dyn 4000 | 1.50 | 4.00 | 2.50 | 6.00 | 1.00 | 7.00 | 4.00 | 6.25 |
| Horiba ABX | ||||||||
| Pentra 60C+ | 2.00 | 3.00 | 2.00 | 7.50 | 1.00 | 7.00 | 5.00 | 5.00 |
| Pentra 120 | 2.00 | 3.00 | 2.00 | 7.50 | 1.00 | 7.00 | 5.00 | 5.00 |
| Pentra XL80 | 2.00 | 3.00 | 2.00 | 7.50 | 1.50 | 7.00 | 5.00 | 5.00 |
| Pentra 80 | 2.00 | 3.00 | 2.00 | 7.50 | 1.00 | 7.00 | 5.00 | 5.00 |
| Pentra DX120 | 2.00 | 3.00 | 2.00 | 7.50 | 1.00 | 7.00 | 5.00 | 5.00 |
| Bayer | ||||||||
| Advia 120 | 1.20 | 5.00 | 2.70 | 5.50 | 0.93 | 7.53 | 2.93 | 8.53 |
| Advia 70 | 1.20 | 5.00 | 2.00 | 7.50 | 1.00 | 7.00 | 3-10 | 2.50 |
| Advia 2120 | 1.20 | 5.00 | 2.70 | 5.50 | 0.93 | 7.53 | 2.93 | 8.53 |
| Beckman | ||||||||
| Coulter LH700 | 0.80 | 7.50 | 1.70 | 8.82 | 0.80 | 8.75 | 3.30 | 7.57 |
| LH 780 | 0.80 | 7.50 | 1.70 | 8.82 | 0.80 | 8.75 | 3.30 | 7.57 |
| LH 750/LH 755 | 0.80 | 7.50 | 1.70 | 8.82 | 0.80 | 8.75 | 3.30 | 7.57 |
| Gen-S System | 0.80 | 7.50 | 1.70 | 8.82 | 0.80 | 8.75 | 3.30 | 7.57 |
| Coulter LH500 | 2.00 | 3.00 | 2.50 | 6.00 | 1.50 | 4.66 | 5.00 | 5.00 |
| Coulter HMX | 2.00 | 3.00 | 2.50 | 6.00 | 1.50 | 4.66 | 5.00 | 5.00 |
| Coulter LH1500 | 0.80 | 7.50 | 1.70 | 8.82 | 0.80 | 8.75 | 3.30 | 7.57 |
| Ac*T5 diff family Ac*t 5diffA1C |
2.00 | 3.00 | 2.00 | 7.50 | 1.00 | 7.00 | 5.00 | 5.00 |
| Sysmex | ||||||||
| XE-2100 | 1.50 | 4.00 | 3.00 | 5.00 | 1.00 | 7.00 | 4.00 | 6.25 |
| XE-2100D | 1.50 | 4.00 | 3.00 | 5.00 | 1.00 | 7.00 | 4.00 | 6.25 |
| XE-2100L | 1.50 | 4.00 | 3.00 | 5.00 | 1.00 | 7.00 | 4.00 | 6.25 |
| XE-AlphaN HST-N | 1.50 | 4.00 | 3.00 | 5.00 | 1.00 | 7.00 | 4.00 | 6.25 |
| XT-2000i | 1.50 | 4.00 | 3.00 | 5.00 | 1.50 | 4.66 | 4.00 | 6.25 |
| XT-1800i | 1.50 | 4.00 | 3.00 | 5.00 | 1.50 | 4.66 | 4.00 | 6.25 |
Overall, these data are encouraging. We see some Sigma metrics at or higher than Six Sigma - excellent news. The ideal goal is not far away.
However, remember these are metric estimates based on an assumption of zero bias. We made these metric calculations soley on the basis of the precision claim and the quailty requirements.
What are the possible consequences of these Sigma metrics?
Now that we have some metrics, to what practical use can we put them? One exercise we can perform is to go through QC Design (or QC Planning) phase with the data and metrics that we have in hand. That is, we can pick the control rule and number of controls that we would need to use with each run.
There are a variety ot QC Design tools available for your use. See below for the options.
Please note that the exercise below uses the "worst performer" as the basis for QC Design. Assuming that all the different methods will have the same number of controls run and same control rules applied, then the safest way to plan QC for the entire instrument is to work with the worst performing method. If you plan QC for that, then you have assured the quality for all the methods on the instrument. (There are other schools of thought on how to plan QC on a multitest instrument, but for this application, we take a rigorous and conservative approach.)
Below follows the results of QC Design:
| Analyte | RBC | WBC | Hb | Pt |
Control Rule
|
Error |
| Erthrocytes | leukocytes | Hemoglobin | Platelet Count |
for
|
Detection | |
| Total Allowable Error % | +/- 6% | +/- 15% | +/- 7% | +/- 25% |
Worst Performer
|
Estimated |
| Instrument | Sigma Estimate | Sigma Estimate | Sigma Estimate | Sigma Estimate | ||
| Abbott | ||||||
| Cell-Dyn Ruby | 3.33 | 6.25 | 5.00 | 6.58 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.53
|
| Cell-Dyn Sapphire | 4.00 | 5.50 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| Cell-Dyn3200 | 4.00 | 5.50 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| Cell-Dyn3700 | 4.00 | 6.00 | 5.80 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| Cell-Dyn4000 | 4.00 | 6.00 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| Horiba ABX | ||||||
| Pentra 60 | 3.00 | 7.50 | 7.00 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Pentra 120 | 3.00 | 7.50 | 7.00 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Pentra XL80 | 3.00 | 7.50 | 7.00 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Pentra 80 | 3.00 | 7.50 | 7.00 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Pentra DX120 | 3.00 | 7.50 | 7.00 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Bayer | ||||||
| Advia 120 | 5.00 | 5.50 | 7.53 | 8.53 | 1:3s N=3 |
0.91
|
| Advia 70 | 5.00 | 7.50 | 7.00 | 2.50* | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.11
|
| Advia 2120 | 5.00 | 5.50 | 7.53 | 8.53 | 1:3s N=3 |
0.91
|
| Beckman | ||||||
| Coulter LH700 | 7.50 | 8.82 | 8.75 | 7.57 | 1:3.5s N=3 |
>0.96
|
| LH 780 | 7.50 | 8.82 | 8.75 | 7.57 | 1:3.5s N=3 |
>0.96
|
| LH 750/ Lh 755 | 7.50 | 8.82 | 8.75 | 7.57 | 1:3.5s N=3 |
>0.96
|
| Gen-S System | 7.50 | 8.82 | 8.75 | 7.57 | 1:3.5s N=3 |
>0.96
|
| Coulter LH500 | 3.00 | 6.00 | 4.66 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Coulter HMX | 3.00 | 6.00 | 4.66 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Coulter LH1500 | 7.50 | 8.82 | 8.75 | 7.57 | 1:3.5s N=3 |
>0.96
|
| Ac*T5 diff family Ac*t 5diffA1C |
3.00 | 7.5 | 7.00 | 5.00 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.32
|
| Sysmex | ||||||
| XE-2100 | 4.00 | 5.00 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| XE-2100D | 4.00 | 5.00 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| XE-2100L | 4.00 | 5.00 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| XE-AlphaN HST-N | 4.00 | 5.00 | 7.00 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| XT-2000i | 4.00 | 5.00 | 4.66 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
| XT-1800i | 4.00 | 5.00 | 4.66 | 6.25 | 1:3s/2of3:2s/R:4s/3:1s N=3 |
0.84
|
*Bayer provided an estimate of 3-10% precision. 10% was used for the calculations.
It's immediately clear that most of these instruments require using "Westgard Rules" (multirule combinations) and 3 controls per run, because of the worst performer. This, based on the optimistic claim and assumption of no bias. Clearly, most of these analyzers aren't ready to perform the CLIA minimums. Also, we should note that we set the practical maximum control rule recommendation at "Westgard Rules" with 3 controls per run - even in cases where more QC is needed, it's unlikely that a lab will be willing to run 6 controls per run.
We've also added some quantified error detection estimates. The numbers are good to within +/- a few percent. The ideal goal that we usually set for error detection is 90% or more. The differences noted in error detection show the major difference between a Sigma metric of 4.00 (84% error detection, which is pretty good) and a Sigma metric of 2.5 (11% error detection, not so good). Even though both method may be using "Westgard Rules", the effectiveness of that monitoring will differ. In the former case, using "Westgard Rules" can provide the necessary error detection. In the latter case, there aren't enough rules to provide the error detection needed.
What do those error detection numbers mean, in practical terms? You can convert them into Average Run Length figures to determine how long it will take for the QC procedure to detect a problem, when a problem occurs. The goal is, of course, to try and detect the problem within the run where it occurred. Here's the equation: ARL = 1/(error detection). So for the QC procedure with 0.84 error detection, it will take 1/(0.84) = 1.19 runs to detect the problem. Basically you will catch the problem most of the time within the run where the problem occurs, with a few cases where it will take 2 runs to detect the problem.
Observations, Conclusions, and Important Notes.
1. Sigma Metrics Estimates are still estimates.
The real metric that matters is how the method performs in your laboratory. Getting more information is critical to your decision process. Method Validation results, ideally from studies that you run in your lab, will provide you with the best information.
With Sigma metrics calculated from performance claims, you may be able to better recognize poor performance and eliminate those methods from your consideration.
2. CLIA minimums are not enough.
Running just 2 controls per run would not be sufficient for an overwhelming majority of these instruments. Using "Westgard Rules" and only 2 controls on a 3.0 Sigma method would only provide approximately 14% error detection (ARL of 7 runs before you would detect the problem). Some of these instruments even look like they need 6, 8, or more controls per run!
3. What would you do?
Undoubtedly, the precision claim is not the only factor that goes into the decision to purchase an instrument. But it is worth wondering, how important is the. If you know that a method will require you to run 4 or more controls and use "Westgard Rules", will that affect your decision. Will you accept poor method performance in favor of speedy test results? Let us know your thoughts.