Sigma Metric Analysis

What happens when you measure quality of a small wet chemistry veterinary analyzer using not only QC, but also pooled samples of canines and felines? Is this analyzer safe for any species?

Sigma-metric Analysis of KeyLab Veterinary chemistry analyzer

Sten Westgard, MS
December 2017

• The Precision, Comparison and Sigma-metric data
• Summary of Performance according to CLIA Goals by Sigma-metrics Normalized Method Decision charts
• Conclusion

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

In a previous Sigma-metric analysis, we leveraged a veterinary study to draw some conclusions about an instrument's fitness for purpose for humans. In this analysis, we're going to take a veterinary chemistry analyzer and focus solely on its use in animals, specifically canines and felines.

Katherine L. Irvine, Kay Burt, Kostas Papasouliotis, Evaluation of an in-practice wet-chemistry analyzer using canine and feline serum samples  J Vet Diagn Invest 2016 28(1):38-45.

This study looks at a small analyzer that would be used within a clinic setting. In addition to running the usual controls to establish precision, the study authors also create pooled serum samples from canines and felines, allowing for additional species-specific evaluations:

"Blood samples from 66 dogs and 59 cats submitted to the Diagnostic Laboratories (University of Bristol, Langford, Bristol,UK) between March 2013 and March 2014 were included in the study.... Excess serum from clinical samples following analysis was frozen at -20C for up to 1 year and used in the study with owners' consent.... Pooled serum samples were also created for both species using excess serum from stored clinical samples. For simplicity, a single pooled sample from each species was used rather than multiple samples with different  concentrations for all of the different analytes. Canine and feline pooled samples were created by mixing several samples containing within- or near-reference concentrations for all analytes in this study, as determined by the reference chemistry analyzer."

The Imprecision, Bias and Sigma-metric Data

"Two human-derived QCM (Eurocontrol N and Eurocontrol P) were run daily on the in-practice analyzer....The imprecision of the in-practice analyzer was assessed by within- and between-run repeatability using the QCM...and the canine and feline serum pools. Within-run repeatability was determined by measuring each analyte in the same sample 10 times sequentially within the same assay run. Between-run repeatability using the serum pools was determined by measuring each analyte in the same pool each day for 5 consecutive days. Between-run repeatblity using the QCM was determined from the daily control values generated over a 2-month period (10 measurements)."

"The bias was determined by calculating the mean bias from EQA reports for 6 months for the control material most closely approximating the level of clinical significance with the most stringent quality requirement...."

So for the QC materials, we have two levels measured for each analyte. Notice the column for TEa lists the specific goals for veterinary use. These are different than the usual CLIA, RCPA, or "Ricos Goals".

And here are the canine serum pool numbers:

 Finally, here are the feline serum pool metrics:

Wow, that's a huge amount of numbers. What do they all mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the KeyLab Vet  Analyzer

Remember the equation for the Sigma metric is (TEa - bias) / CV:

For a 15% quality requirement for albumin on the KeyLab, the equation is (15 -2) / 2.16 = 6.02

The metrics are displayed along the right column.

This is a mixed bag of performance. Some very high Sigma-metrics for some assays, whereas very low Sigma-metrics for others. A few cases where the bias exceeds the allowable total error.

Remember this study did not one, but two, very unique things. They built serum pools for both canines and felines.

So we have two more sets of Sigma-metrics. Here are the Sigma-metrics for canines:

You can see very quickly that there are significant biases between the canine samples measured on the KeyLab and the core laboratory. There's also an extremely low CV for cholesterol that produces an extremely high Sigma-metric, but that metric seems a little suspicious (maybe a typo, recording error, or outright fluke?).

Finally, here are the feline Sigma-metrics.

There are even more analytes here with negative Sigma-metrics.

Summary of Performance by Sigma-metrics Method Decision Chart using ASVCP Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

 There are four levels here that have completely missed the target, with less than 2 Sigma performance. This would be considered bad, but when we look at the canine pool, the story gets considerably worse.

And here is the feline chart:

The feline pool has more tests missing the target than hitting it. Traditionally, we operate under the assumption that our control materials are essentially the same as patient samples. But here we see the opposite. It looks like the QC materials have a difference verdict on the fitness of this instrument than what the feline and canine pools are indicating. It appears that the patient pools are finding that there are significant biases between this instrument and its peer instruments.

Normally at this stage, we would try to estimate what level of QC effort would be required for the laboratory if indeed the instrument was installed. However, with three different verdicts, two of them pretty bad, with one of them questionable, the QC is less of a question, and the bigger question is whether this should be installed in any lab at all.

There is also the further challenge that most small clinic analyzers have very limited, even primitive, QC capabilities. So if we were able to determine that this instrument should be running full "Westgard Rules" with 4 or 8 controls per run, we might be very disapointed to discover that the instrument is only capable of implementing 2 SD limits and can only run 2 or 3 controls at a time. The smaller devices are often not only challenged in their performance, they are also limited in their QC features.

Conclusion: Unsafe for any species?

The authors stated "The in-practice analyzer showed good overall performance, with only calcium and phosphate analytes identified as significantly problematic."

We would reach a different conclusion based on the data and graphs above. If we believe the canine and feline serum pools, we have an instrument which is not fit for purpose for many of these assays. Most of these assays are "significantly problematic" when we use the Sigma scale. Performance this poor will generally require multiple tests,  additional consults, more time spent visiting the vet, and etc.