Tools, Technologies and Training for Healthcare Laboratories

Mid-Volume Chemistry analyzer 2009

We take a look at data on a new mid-volume chemistry analyzer, based on data from an abstract at the 2009 AACC/ASCLS/CSCC meeting. If you buy the latest generation of chemistry instruments, does that guarantee you're getting world class methods?

June 2009
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

It's time again for an anonymous QC application. There's a poster in the 2009 AACC/ASCLS with method performance data for a new analyzer for the mid-volume laboratory. We're not naming names because, well, you'll see.

The study covers Urea Nitrogen (BUN), IgG, Na, Potassium (K), and Chloride (Cl). These are analytes with some of the tightest, toughest analytical quality requirements.

The Precision and Comparison data

The precision data was given for two levels of control

Analyte Level CV%
BUN 15 3.4
69 2.8
IgG 7 2.5
13.6 2.6
Na 137 1.2
149 1.3
K 4 1.2
7 1.3
Cl 1.88 2.11
120 1.4

Nothing extraordinary so far. The experienced laboratory professional might be able to make some initial judgments about the tests (are those CVs higher than your current methods? or lower?) But without some more data, it's hard to know whether or not this is good performance.

Next, we have the comparison study data. This was done using patient samples:

Analyte Slope Y-Intercept Correlation Coefficient N
BUN 1.04 0.39 1.0 45
IgG 0.99 0.16 1.0 46
Na 0.97 0.46 0.96 47
K 1.01 -0.09 1.0 48
Cl 0.96 1.71 0.99 47

Many people will focus on the correlation coefficient, assuming that a value near 1.0 means everything is fine. Actually, the correlation coefficient simply tells us whether or not the spread of the data is wide enough that we can use linear regression. The values are indeed good, so linear regression is sufficient.

Having this comparison data is good, but not complete. We need to calculate bias at the same levels where we have the imprecision data. We will use the slope and y-intercept to calculate the bias.

For example, with BUN, at a level of 15 mg/mL, the test method value will be y = (15 * 1.04) + 0.39 = 15.6 + 0.39 = 15.99

The bias at this level is [ (15.99 - 15) / 15 ] * 100 = [ 0.99 / 15 ] * 100 = 6.6%

Analyte Slope Y-Intercept Level % Bias
BUN 1.04 0.39 15 6.6%
69 4.56%
IgG 0.99 0.16 7 1.28%
13.6 0.17%
Na 0.97 0.46 137 2.66%
149 2.69%
K 1.01 -0.09 4 1.25%
7 0.29%
Cl 0.96 1.71 101 2.31%
120 2.58%

Determine Quality Requirements at the decision level

Now that we've got our imprecision and inaccuracy data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement.

For all the analytes covered in this application, there are existing CLIA analytical quality requirements. We can look those up easily. However, some of those quality requirements are actually multiple goals for the same test. For each level, we may need to calculate the specific quality required by the test.

For example: with BUN, the CLIA quality requirement is +/- 2 mg/dL or =+/- 9%, whichever is greater

At the lower level of 15, 2/15 is a quality requirement of 13.33%.

At the higher level of 69 mg/dL, the quality requirement is 9%.

Below is a calculation of all the quality requirements we need:

 

Analyte Level Quality Requirement Type Calculations Quality Requirement %
BUN 15 Target value ± 2 mg/dL or ± 9%,
whichever is greater
2/15 13.33%
69 (2/69) < 9% 9%
IgG 7 ± 25% none 25%
13.6 none 25%
Na 137 Target value ± 4 mmol/L 4/137 2.92%
149 4/149 2.68%
K 4 Target value ± 0.5 mmol/L 0.5/4 12.5%
7 0/5/7 7.14%
Cl 1.88 Target value ± 5% none 5%
120 none 5%

Here is where we see that these particular analytes have very small targets. These are truly some of the most demanding quality requirements in the laboratory.

Calculate Sigma metrics

Now we have all the pieces in place. Since we have two levels of data, we will calculate two Sigma-metrics. Note that you may decide only one of those levels is the most important, and optimize your design around performance at that level.

Remember the equation for Sigma metric is (TEa - bias) / CV:

At the lower level, Sigma is, (13.33 - 6.6) / 3.4 = 1.98

The metrics are displayed along the right column.

Analyte Slope Y-Intercept Level % Bias %CV Quality Requirement Sigma-metric
BUN 1.04 0.39 15 6.6% 3.4% 13.33% 1.98
69 4.56% 2.8% 9.0% 1.58
IgG 0.99 0.16 7 1.28% 2.5% 25.0% 9.49
13.6 0.17% 2.6% 25.0% 9.55
Na 0.97 0.46 137 2.66% 1.2% 2.92% 0.21
149 2.69% 1.3% 2.68% negative
K 1.01 -0.09 4 1.25% 1.2% 12.50% 9.38
7 0.29% 1.3% 7.14% 5.27
Cl 0.96 1.71 101 2.31% 1.0% 5% 2.69
120 2.58% 1.4% 5% 1.73

A pretty wide range of performance here. We've got some analytes that perform at world class quality (IgG), while others are below the acceptable level for quality.

Summary of Performance by Sigma-metrics chart

Here's a Method Decision chart, using Six Sigma metrics lines to delineate the performance of the methods.

qcapp58f1

Those familiar with a Method Decision chart know that the various lines indicate Sigma threshholds of performance. For instance, the region closest to the origin represents "world class" or Six Sigma performance. IgG and K are in or are close to that region. Contrast this to the other analytes, where both imprecision and inaccuracy is excessive.

QC Design using OPSpecs chart

Not only can we use tools to graphically depict the performance of the method - we can also use those tools to help determine the best QC procedure to use with that method. Using EZ Rules 3, we can express the method performance on an OPSpecs (Operating Specifications) chart :

qcapp58f2

Normalized OPSpecs chartThe news here is mixed. IgG and K can be very easily controlled. However, Na, CL and BUN are uncontrollable. Even if we had the full "Westgard Rules" implemented, that would not be enough power to bring those methods under control. Other non-statistical QC techniques will need to be used in order to keep those methods in line.

Conclusion

This 2009 study shows that purchasing just any of the latest chemistry instruments (5th, 6th generation) will not guarantee world class quality. This instrument here has some good methods and some bad methods.