Tools, Technologies and Training for Healthcare Laboratories

Sigma-metric Analysis of an AU 5800 in Turkey

At the end of December 2016, the Turkish Clinical Biochemistry journal published a study looking at the month to month Sigma-metrics of an AU 5800. Are Sigma-metrics changing month to month on this analyzer? What does that mean for QC?

Sigma-metric Analysis of an AU 5800 in Turkey

Sten Westgard, MS
June 2017

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

This analysis uses data from a recent study in Turk Klinik Biyokimya Derg:

Dilke Iren Emekli, Kenan Guclu, Gamze Turna, Bilal Ilanbey, Rukiye Nar, Esin Avc Cicek, Evaluation of Analytical Process in Clinical Laboratories: Six Sigma Methodology Turk Klinik Biyokimya Derg 2016; 14(2): 79-86.

The unique part of this study, beyond the fact that it looks at performance in a Turkish hospital, is that they assessed Sigma-metrics for three months from July through October 2014.

The Imprecision, Bias and Sigma-metric Data

"Internal quality control data between July-October 2014 werre obtained from the laboratory information system."

 Two levels of controls were measured, but we further concentrated the Sigma-metric assessment on just one decision level per test, which is highlighted in yellow.

Turkish AU 5800 July   Instrument Information
TEST TEa Source TEa% Group or PeerMean Lab Mean Bias Units % Bias SD CV Sigma
Albumin  CLIA 10 4.72 4.70 0.02 0.4 0.10 2.13 4.50
  10 2.18 2.28 0.10 4.6 0.05 2.19 2.47
Alk Phos
 
CLIA 30 131.00 119.00 12.00 9.2 5.10 4.29 4.86
  30 522.00 471.00 51.00 9.8 16.00 3.40 5.96
ALT  CLIA 20 43.60 42.00 1.60 3.7 0.89 2.12 7.71
  20 128.00 121.00 7.00 5.5 3.30 2.73 5.33
AST
 
CLIA 20 48.50 49.00 0.50 1.0 1.30 2.65 7.15
  20 138.00 137.00 1.00 0.7 3.70 2.70 7.14
Creatinine Kinase (CK)
 
CLIA 30 167.00 164.00 3.00 1.8 3.40 2.07 13.60
  30 382.00 385.00 3.00 0.8 11.00 2.86 10.23
Creatinine  CLIA 15 1.30 1.30 0.00 0.0 0.06 4.62 3.25
  15 5.11 4.70 0.41 8.0 0.30 6.38 1.09
GGT  Ricos 22.11 57.90 55.00 2.90 5.0 0.80 1.45 11.76
  22.11 158.00 155.00 3.00 1.9 3.70 2.39 8.47
Glucose  CLIA 10 100.00 100.00 0.00 0.0 3.00 3.00 3.33
  10 239.00 241.00 2.00 0.8 6.00 2.49 3.68
Protein, Total  CLIA 10 3.64 3.60 0.04 1.1 0.10 2.78 3.20
  10 7.92 7.80 0.12 1.5 0.20 2.56 3.31
Urea Nitrogen  CLIA 9 39.20 38.30 0.90 2.3 0.60 1.57 4.28
  9 161.00 159.00 2.00 1.2 4.00 2.52 3.08

 

Turkish AU 5800 August   Instrument Information
TEST TEa Source TEa Group or PeerMean Lab Mean Bias Units % Bias SD CV Sigma
Albumin  CLIA 10 4.72 4.60 0.12 2.5 0.12 2.61 2.86
  10 2.18 2.10 0.08 3.7 0.05 2.38 2.66
Alk Phos  CLIA 30 131.00 126.00 5.00 3.8 9.00 7.14 3.67
  30 522.00 498.00 24.00 4.6 22.00 4.42 5.75
ALT  CLIA 20 43.60 41.00 2.60 6.0 0.64 1.56 8.99
  20 128.00 118.00 10.00 7.8 3.80 3.22 3.78
AST  CLIA 20 48.50 48.00 0.50 1.0 1.21 2.52 7.52
  20 138.00 132.00 6.00 4.3 5.00 3.79 4.13
Creatinine Kinase (CK)  CLIA 30 167.00 160.00 7.00 4.2 2.70 1.69 15.29
  30 382.00 371.00 11.00 2.9 9.00 2.43 11.18
Creatinine  CLIA 15 1.30 1.30 0.00 0.0 0.05 3.85 3.90
  15 5.11 5.00 0.11 2.2 0.17 3.40 3.78
GGT  Ricos 22.11 57.90 55.00 2.90 5.0 0.91 1.65 10.34
  22.11 158.00 153.00 5.00 3.2 4.00 2.61 7.25
Glucose  CLIA 10 100.00 96.10 3.90 3.9 1.88 1.96 3.12
  10 239.00 229.00 10.00 4.2 6.00 2.62 2.22
Protein, Total  CLIA 10 3.64 3.50 0.14 3.8 0.09 2.57 2.39
  10 7.92 7.70 0.22 2.8 0.19 2.47 2.93
Urea Nitrogen  CLIA 9 39.20 38.00 1.20 3.1 0.80 2.11 2.82
  9 161.00 155.00 6.00 3.7 4.00 2.58 2.04

 

Turkish AU 5800 SEPTEMBER 
Instrument Information
TEST TEa Source TEa% Group or PeerMean Lab Mean Bias Units % Bias SD CV Sigma
Albumin CLIA 10 4.72 4.70 0.02 0.4 0.10 2.13 4.50
    10 2.18 2.10 0.08 3.7 0.04 1.90 3.32
Alk Phos  CLIA 30 131.00 126.00 5.00 3.8 4.10 3.25 8.05
  30 522.00 500.00 22.00 4.2 19.00 3.80 6.79
ALT  CLIA 20 43.60 41.00 2.60 6.0 0.86 2.10 6.69
  20 128.00 119.00 9.00 7.0 3.50 2.94 4.41
AST  CLIA 20 48.50 48.00 0.50 1.0 0.81 1.69 11.24
  20 138.00 132.00 6.00 4.3 3.00 2.27 6.89
Creatinine Kinase (CK)  CLIA 30 167.00 162.00 5.00 3.0 3.03 1.87 14.44
  30 382.00 370.00 12.00 3.1 9.00 2.43 11.04
Creatinine  CLIA 15 1.30 1.30 0.00 0.0 0.02 1.54 9.75
  15 5.11 5.00 0.11 2.2 0.12 2.40 5.35
GGT  Ricos 22.11 57.90 55.00 2.90 5.0 0.73 1.33 12.88
  22.11 158.00 154.00 4.00 2.5 4.00 2.60 7.54
Glucose  CLIA 10 100.00 99.40 0.60 0.6 2.60 2.62 3.59
  10 239.00 236.00 3.00 1.3 6.00 2.54 3.44
Protein, Total  CLIA 10 3.64 3.60 0.04 1.1 0.12 3.33 2.67
  10 7.92 8.00 0.08 1.0 0.20 2.50 3.60
Urea Nitrogen  CLIA 9 39.20 39.10 0.10 0.3 1.07 2.74 3.20
  9 161.00 161.00 0.00 0.0 3.00 1.86 4.83

 Yes, that is a whole lot of numbers! What do they all mean? In the absence of context, it's often hard to know.

So let's apply Sigma-metrics and plot the performance visually.

Summary of Performance by Sigma-metrics Method Decision Chart using CLIA Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

2017 AU 5800 Turkey July NMEDX

 

2017 AU 5800 Turkey August NMEDX

 

2017 AU 5800 Turkey September NMEDX

 2017 AU 5800 Turkey Three Month Sigmas

We can see some considerable variation in the month to month Sigma-metric of these assays on the AU 5800. Some methods are consistently poor in performance, such as Albumin and Total Protein. But others will vary from one Sigma category to another depending on the month. August was a poor month, but September saw a marked rise in most of the metrics. For Alkaline Phosphatase, it varies month to month from 4 Sigma, down to 3 Sigma, then up beyond 6 Sigma.

Summary of QC Design by Normalized OPSpecs chart - using CLIA Goals

2017 AU 5800 Turkey July NOPSpecs

2017 AU 5800 Turkey August NOPSpecs

2017 AU 5800 Turkey September NOPSpecs

There are many QC rules that need to be implemented on this AU 5800 instrument, from just wide 3s limits and 2 controls, to the full "Westgard Rules" and 8 controls. The AU 5800 will need to keep an eye on several analytes but there are several analytes that are going to be quite easy to control. For a test like alkaline phosphatase, the QC would need to be changed from the almost the full "Westgard Rules" at 4 Sigma, to the full "Westgard Rules" and increased controls being run for 3 Sigma, and then dial back to just 3 SD control limits and 2 controls per run once the Sigma rises to Six. That's quite a bit of change from month to month on just one analyte. If this is a consistent bounce across more than the 3 months in this study, it means the method and/or the laboratory has some consistency of quality issues. A constantly shiftly QC strategy for these methods will be tough to implement for most laboratories.

 

Conclusion

The authors stated "The study showed that tests with low process sigma levels in our laboratory can be determined and these tests may be evaluated as a whole with preanalytical and postanalytical processes. Six sigma methodology may provide a detailed assessment of measurement processes with problematic analytical process sigma levels and controlling the variables." 

This seems to be a charitable description of what the data tells us. Around half of those assays don't need the "Westgard Rules" because performance is so good. But there are some problematic assays, which will require an intensified QC effort.

The wisdom of calculating Sigma-metrics from month to month is another topic. In the Sigma VP program and in most of our general recommendations for implementing Sigma-metrics, we encourage laboratories that the ideal metrics come from data accumulated over a period of 3 to 6 months. For highly stable instruments and analytes, the month to month Sigma-metrics are overkill. But for methods that are not so stable, it may be necessary to benchmark them more frequently, even if that delivers unfortunate news about the performance.