Tools, Technologies and Training for Healthcare Laboratories

Quality Goal Index

Dr. David Parry of St. Boniface General Hospital in Winnipeg, provides us with data on Sigma metrics in two laboratories. Dr. Parry's innovation is a Quality Goal Index, a metric that can distinguish between precision and accuracy problems, as well as techniques to deal with calibrator lot changes.

Its Use in Benchmarking and Improving Sigma Quality Performance of Automated Analytic Tests

The long-term goal of six sigma quality management is to achieve an error rate of 3.4 or less per million opportunities for all laboratory processes. In percent terms, that’s an error rate of less than 0.001%. This very demanding goal is achievable or within reach for many automated analytic processes, but implicitly requires knowledge of which processes fall short of this goal and why. Hence, a systematic approach to benchmarking sigma quality performance of analytic processes is fundamental to practicing six sigma quality management. In order to achieve six sigma quality, it is necessary to assess and document which analytical processes do and do not achieve six sigma quality performance and to understand why.

Sigma Quality Performance

As discussed elsewhere (1), quality performance of any analytical process in a laboratory can be described in terms of its sigma metric. Calculation of sigma performance is quite simple:

Sigma = (TEa – Bias) / CV

The harder part is making sure that data used in this calculation really reflects current analytical performance. In my region, we turned to our on-analyzer quality control databases for estimates of imprecision (CV). This data reflects actual analytical imprecision, if and only if, data integrity has been maintained. Data integrity can be assured only when procedures are in place and rigorously practiced to exclude erroneous quality control results due to procedural blunders and statistical outliers. This does not mean excluding out-of-control results; no statistically-valid data should be edited out.

For estimates of test bias, our region makes use of Randox Interlaboratory Quality Assessment Scheme (RIAQS). This external quality assessment program, run out of Ireland, provides biweekly (every 2 weeks) challenges for a broad range of automated chemistry tests. There are two main attributes that make this program of particular value for assessing test bias. First, challenges are blind and second, they are at varying analyte concentrations. These two attributes, combined with its biweekly frequency and relatively large user base, make this database reflective, as close as possible, of “real-world” testing bias. Bias is calculated for each test application based on the average (running mean) bias of the last 10 values obtained over the preceding five month period, updated every two weeks.

Sigma quality management forces one to think in terms of quality needs, not just performance capabilities and this is good since it focuses attention on what really matters. The next step is setting quality goals. This in itself can be a real eye-opener and a little daunting. There are multiple sources for quality goals but the proviso, of course, is objectivity in selecting these goals in deference to analytic capabilities. We selected quality goals for total error (TEa) based on either biological variation or proficiency testing target data accessible through this website (2, 3), fully recognizing that at least some of these goals may need customizing to specific clinical requirements as our experience with this approach develops. Selecting test specific goals, as uncomfortable or imperfect a process as this may be, it reinforces the fact that good clinical practice is implicitly linked to quality of laboratory testing.

Once current and reliable data for imprecision and bias has been obtained and quality goals set, sigma quality performance is then calculated for each test application. These sigma metrics are date documented and used to benchmark existing sigma performance for inter-instrument and inter-laboratory comparisons within our region and for future historical reviews to assess performance changes.

To illustrate this approach, the sigma metrics for 60 automated test applications over three analytic platforms at one laboratory (site A) in our region indicates that twenty eight (46%) of these applications fall short of meeting six sigma quality performance. Of these, twelve fail to meet minimum sigma quality performance with metrics less than three and another four just meet minimal acceptable performance with sigma metrics between three and four. At another laboratory (site B) in our regional group (seven sites in total), the original data collected indicates more than half (57%) of the automated test applications do not achieve six sigma quality performance.

Quality Goal Index

It must be kept in mind that six sigma quality management is not only a tool for defining process performance but also a method for moving a process from its present error rate to a very low error rate. In order to achieve quality improvement of automated analytic tests where needed, it is important to understand the test-specific reasons for their quality shortcomings, be it either excessive imprecision, excessive bias or both. To facilitate this, performance data is assessed by calculating the quality goal index (QGI) by the following expression:

QGI = Bias / 1.5 CV

Derivation of this expression is based on mathematical reduction of the ratio (Bias/Accuracy Goal) / (CV/Precision Goal). The QGI ratio represents the relative extent to which both bias and precision meet their respective quality goals. The quality goals chosen for use in this expression are 1.5*TEa/6 for bias and TEa/6 for precision based on their widespread use in six sigma methodology literature. For those who feel that inclusion of a1.5 sigma shift in calculating the bias goal is not justified, then both quality goals can be calculated as TEa/6. The corresponding expression would then reduce to QGI = Bias/CV.

Examination of the unreduced equation gives better insight into how this data-evaluating tool works. For example, QGI is higher for a test application when bias exceeds its accuracy goal and imprecision meets its precision goal and QGI is lower when bias meets its accuracy goal and imprecision exceeds its precision goal. The criteria we use for interpreting QGI when test applications fall short of six sigma quality is as follows:

QGI Problem
<0.8 Imprecision
0.8 - 1.2 Imprecision & Inaccuracy
>1.2 Inaccuracy

This quantifiable approach to problem assessment is computer programmable in Microsoft Excel using Visual Basic for Applications (VBA). Macro programming simplifies its application which is particularly helpful when dealing with larger numbers of tests at multiple lab sites. As an example of this approach, QGI indicates that of the 29 test applications that failed to meet six sigma quality performance at lab site A in our region, the main problem is excessive imprecision in 52%, with excessive inaccuracy occurring in 17%. At lab site B, the main problem is excessive inaccuracy in 73% with excessive imprecision being the problem in 20%. QGI provides easy insight into where improvement is required and can serve as a tool for focusing efforts on sigma quality improvement of automated analytic tests.

A complete listing of test applications, sigma metrics, and QGI results: Test Laboratory A:

APPLICATION

INSTUMENT

BIAS%

CV%

QG%

SIGMA

QGI

PROBLEM

Albumin

PPE-P1

1.40

3.40

10

2.53

0.27

Imprecision

Albumin

MOD-2

4.28

1.04

10

5.50

2.74

Inaccuracy

Alkaline Phosphatase

PPE-P1

-14.47

6.50

30

2.39

1.48

Inaccuracy

Alkaline Phosphatase

MOD-2

-13.03

6.00

30

2.83

1.45

Inaccuracy

ALT

PPE-P1

9.17

2.20

32.1

10.42

2.78

None

ALT

MOD-2

8.59

2.00

32.1

11.76

2.86

None

AST

PPE-P1

3.89

2.30

15.2

4.92

1.13

Inaccuracy/Imprecision

AST

MOD-2

0.87

1.30

15.2

11.02

0.45

None

Ammonia

PPE-P1

0.00

3.86

20

5.18

0.00

Imprecision

Ammonia

MOD-2

0.00

6.20

20

3.23

0.00

Imprecision

Total CO2

PPE-P1

0.22

1.62

10

6.04

0.09

None

Total CO2

MOD-2

4.62

2.90

10

1.86

1.06

Inaccuracy/Imprecision

Bilirubin, Direct

PPE-P1

3.99

4.90

48.5

9.08

0.54

None

Bilirubin, Direct

MOD-2

-7.65

2.00

48.5

20.43

2.55

None

Bilirubin, Total

PPE-P2

1.24

1.89

9.5

4.37

0.44

Imprecision

Bilirubin, Total

MOD-2

-3.84

2.20

9.5

2.57

1.16

Inaccuracy/Imprecision

Calcium

PPE-P1

-3.03

1.77

7.6

2.58

1.14

Inaccuracy/Imprecision

Calcium

MOD-2

-2.13

1.51

7.6

3.62

0.94

Inaccuracy/Imprecision

Chloride

PPE-P1

-0.43

1.10

5

4.15

0.26

Imprecision

Chloride

MOD-2

-0.63

1.80

5

2.43

0.23

Imprecision

Cholesterol

PPE-P1

-2.62

1.35

9

4.73

1.29

Inaccuracy

Cholesterol

MOD-2

1.99

1.29

9

5.43

1.03

Inaccuracy/ Imprecision

Creatine Kinase

PPE-P1

-1.25

1.00

30

28.75

0.83

None

Creatine Kinase

MOD-2

-1.01

1.50

30

19.33

0.45

None

Crearinine

PPE-P1

1.58

2.00

15

6.71

0.53

None

Creatinine

MOD-2

-0.06

1.40

15

10.67

0.03

None

GGT

PPE-P1

-2.21

2.20

25

10.36

0.67

None

GGT

MOD-2

-3.29

2.60

25

8.35

0.84

None

Glucose

PPE-P1

-0.20

1.09

6.3

5.60

0.12

Imprecision

Glucose

MOD-2

-1.47

0.80

6.3

6.04

1.23

None

HDL

PPE-P2

4.97

3.20

30

7.82

1.04

None

HDL

MOD-2

9.90

2.70

30

7.44

2.44

None

Hydroxybutrate

PPE-P1

-3.41

6.87

25

3.14

0.33

Imprecision

Iron

PPE-P1

6.08

2.20

30.7

11.19

1.84

None

Iron

MOD-2

5.05

1.40

30.7

18.32

2.40

None

LDH

PPE-P2

1.44

1.30

11.4

7.66

0.74

None

LDH

MOD-2

-0.27

1.30

11.4

8.56

0.14

None

Lipase

PPE-P2

-3.06

3.50

29.1

7.44

0.58

None

Lipase

MOD-2

-6.83

4.10

29.1

5.43

1.11

Inaccuracy/Imprecision

Lactate

PPE-P2

0.00

1.68

30.4

18.10

0.00

None

Lactate

MOD-2

0.00

0.86

30.4

35.35

0.00

None

Magnesium

PPE-P2

2.31

2.20

25

10.31

0.70

None

Magnesium

MOD-2

-0.98

3.90

25

6.16

0.17

None

Phosphate

PPE-P2

-0.22

1.86

4.3

2.19

0.08

Imprecision

Phosphate

MOD-2

1.38

1.70

4.3

1.72

0.54

Imprecision

Potassium

PPE-P1

-0.08

1.15

5.8

4.97

0.05

Imprecision

Potassium

PPE-P2

-0.08

0.72

5.8

7.94

0.07

None

Potassium

MOD-2

-1.50

1.12

5.8

3.84

0.89

Inaccuracy/Imprecision

Protein

PPE-P2

-0.09

1.80

10

5.51

0.03

Imprecision

Protein

MOD-2

-1.09

1.05

10

8.49

0.69

None

Sodium

PPE-P1

1.09

1.01

2.4

1.30

0.72

Imprecision

Sodium

PPE-P2

1.09

0.89

2.4

1.47

0.82

Inaccuracy/Imprecision

Sodium

MOD-2

-0.53

0.87

2.4

2.15

0.41

Imprecision

TIBC

PPE-P2

-2.57

1.80

30

15.24

0.95

None

Triglycerides

PPE-P2

10.00

1.59

27.9

11.26

4.19

None

Triglycerides

MOD-2

-1.63

3.09

27.9

8.50

0.35

None

Urea

PPE-P2

1.09

0.74

15.7

19.74

0.98

None

Urea

MOD-2

0.04

1.01

15.7

15.50

0.03

None

Uric Acid

PPE-P2

6.57

0.70

11.9

7.61

6.26

None

Uric Acid

MOD-2

5.76

1.10

11.9

5.58

3.49

Inaccuracy

A complete listing of test applications, sigma metrics, and QGI results: Test Laboratory A:

APPLICATION

INSTUMENT

BIAS%

CV%

QG%

SIGMA

QGI

PROBLEM

Albumin

PE-1

4.47

2.51

10.00

2.20

7.48

Inaccuracy

Albumin

PE-2

3.05

2.51

10.00

2.77

5.10

Inaccuracy

Alk Phos

PE-1

-13.84

2.25

30.00

7.18

20.76

None

Alk Phos

PE-2

-12.73

2.25

30.00

7.68

19.10

None

ALT

PE-1

1.37

3.05

32.10

10.08

2.79

None

ALT

PE-2

0.20

3.05

32.10

10.46

0.41

None

AST

PE-1

1.73

1.86

15.20

7.24

2.15

None

AST

PE-2

0.81

1.86

15.20

7.74

1.00

None

Bicarbonate

PE-1

-1.82

6.58

10.00

1.24

7.98

Inaccuracy

Bicarbonate

PE-2

-3.84

6.58

10.00

0.94

16.84

Inaccuracy

Bilirubin, Direct

PE-1

-20.63

4.65

44.50

5.13

63.95

Inaccuracy

Bilirubin, Direct

PE-2

-26.49

4.65

44.50

3.87

82.12

Inaccuracy

Bilirubin, Total

PE-1

0.80

3.22

9.50

2.70

1.72

Inaccuracy

Bilirubin, Total

PE-2

-1.37

3.22

9.50

2.52

2.94

Inaccuracy

Calcium, Total

PE-1

-3.18

2.42

7.60

1.83

5.13

Inaccuracy

Calcium, Total

PE-2

-2.23

2.42

7.60

2.22

3.60

Inaccuracy

Chloride

PE-1

-1.40

1.65

5.00

2.18

1.54

Inaccuracy

Chloride

PE-2

-0.89

1.65

5.00

2.49

0.98

Inaccuracy/Imprecision

Cholesterol

PE-1

1.42

1.82

9.00

4.16

1.72

Inaccuracy

Cholesterol

PE-2

-0.66

1.82

9.00

4.58

0.80

Inaccuracy/Imprecision

CK, Total

PE-1

-4.17

1.93

30.00

13.38

5.37

None

CK, Total

PE-2

-3.77

1.93

30.00

13.59

4.85

None

Creatinine

PE-1

-3.25

2.98

15.00

3.94

6.46

Inaccuracy

Creatinine

PE-2

-1.96

2.98

15.00

4.38

3.89

Inaccuracy

GGT

PE-1

-3.25

2.27

25.00

9.58

4.92

None

GGT

PE-2

-2.48

2.27

25.00

9.92

3.75

None

Glucose

PE-1

1.26

2.10

6.30

2.40

1.76

Inaccuracy

Glucose

PE-2

-0.22

2.10

6.30

2.90

0.31

Imprecision

HDL-C

PE-1

1.69

2.15

30.00

13.17

2.42

None

HDL-C

PE-2

3.45

2.15

30.00

12.35

4.95

None

Hydroxybutrate

PE-1

2.34

3.85

25.00

5.89

6.01

Inaccuracy

Hydroxybutyrate

PE-2

2.04

3.85

25.00

5.96

5.24

Inaccuracy

Iron

PE-1

0.45

2.58

30.70

11.72

0.77

None

Iron

PE-2

3.98

2.58

30.70

10.36

6.85

None

LDH

PE-1

-0.72

1.28

11.40

8.34

0.61

None

LDH

PE-2

0.28

1.28

11.40

8.69

0.24

None

Lipase

PE-1

0.54

4.58

29.10

6.24

1.65

None

Lipase

PE-2

-0.31

4.58

29.10

6.29

0.95

None

Magnesium

PE-1

0.62

2.86

25.00

8.52

1.18

None

Magnesium

PE-2

1.29

2.86

25.00

8.29

2.46

None

Phosphate

PE-1

-0.59

3.23

4.30

1.15

1.27

Inaccuracy

Phosphate

PE-2

1.06

3.23

4.30

1.00

2.28

Inaccuracy

Potassium

PE-1

-1.08

13.7

5.80

0.34

9.86

Inaccuracy

Potassium

PE-2

-0.71

13.7

5.80

0.37

6.48

Inaccuracy

Protein, Total

PE-1

-0.59

1.52

10.00

6.19

0.60

None

Protein, Total

PE-2

-0.14

1.52

10.00

6.49

0.14

None

Sodium

PE-1

-0.63

1.79

2.40

0.99

0.75

Imprecision

Sodium

PE-2

-0.17

1.79

2.40

1.25

0.20

Imprecision

UIBC/TIBC

PE-1

-10.00

8.70

30.00

2.30

58.00

Inaccuracy

UIBC/TIBC

PE-2

-3.24

8.70

30.00

3.08

18.79

Inaccuracy

Triglycerides

PE-1

4.21

2.27

27.90

10.44

6.37

None

Triglycerides

PE-2

-2.35

2.27

27.90

11.26

3.56

None

Urea

PE-1

0.00

2.80

15.70

5.61

0.00

Imprecision

Urea

PE-2

-0.91

2.80

15.70

5.28

1.70

Inaccuracy

Uric Acid

PE-1

5.93

1.62

11.90

3.69

6.40

Inaccuracy

Uric Acid

PE-2

5.31

1.62

11.90

4.07

5.73

Inaccuracy

Summary of Laboratory Performance Problems

SITE

NUMBER OF PERFORMANCE PROBLEMS (PERCENT)

NONE

IMPRECISION

INACCURRACY

BOTH

Lab A

32 (54%)

14 (23%)

5 (8%)

9 (15%)

Lab B

24 (43%)

4 (7%)

26 (46%)

2 (4%)

Sigma Quality Improvement

One cause of difficulty in reducing inaccuracy of automated tests is the variable error introduced when switching from one lot of calibrator to another. The manufacturer of the multi-test calibrator we use claims a variability limit of +/-5%. A tolerance of 10% is not adequate for achieving long-term accuracy for six sigma quality performance of automated analytic tests.

Achieving and maintaining six sigma quality performance requires a rigorous approach to calibrator lot changes, one linked to a bias reference point or “anchor”. To accomplish this, the current biases of our “anchor tests” are incorporated into the decision making process during new calibrator lot evaluation. This concept is easiest to understand when described by the procedural steps involved as follows:

TEST EXAMPLES
Creatinine Urea
1 Obtain assigned value for new calibrator 331.0 15.1
2 Assay new calibrator over a one month period to establish assayed mean value 322.34 15.28
3 Subtract assayed mean value from calibrator assigned value to determine % difference from assigned value 2.62 -1.19
4 Obtain % bias from RIQAS 0.41 1.03
5 Calculate % combined error (% difference + % bias) 3.03 -0.17
6 Calculate accuracy goal (1.5TEa/6) 3.8 3.9
7 Determine ratio % bias / accuracy goal (B/AG) 0.11 0.26
8 Determine ratio % combined error / accuracy goal (CE/AG) 0.89 -0.04
9 Select either assigned value or assayed mean value for new calibrator setpoint depending on which ratio is numerically lower. If B/AG < CE/AG, use assayed mean value. If CE/AG < B/AG, use new calibrator assigned value 322.34 15.1

In the accompanying test examples, the assayed mean value is selected as calibrator setpoint for creatinine, whereas the assigned value is selected for urea. Each of these choices gives best calibrator cross-over accuracy on the RIQAS quality assessment program. Using this approach for twenty-three “anchor tests” at laboratory site A, it was determined that best cross-over accuracy was achieved by using the assigned values for new calibrator setpoints for 12 of these tests (Table 3). Assayed means provided better cross-over accuracy for the remaining 11 tests. In addition, the number of tests exceeding their six sigma accuracy goals decreased from five with the old lot to three with the new lot of calibrator, which is expected to improve further with successive lot changes.

APPLICATION

ASSIGNED VALUE

ASSAYED MEAN

%DIFFERENCE

BIAS%

CE%

AG%

B/AG

CE/AG

NEW SETPOINT

Albumin

28.00

28.44

-1.57

1.77

0.20

2.5

0.71

0.08

28.0

Alkaline Phosphatase

227.00

217.50

4.19

-14.60

-10.41

7.5

-1.95

-1.39

227.0

ALT

113.00

122.39

-8.31

9.88

1.57

8.0

1.23

0.20

113.0

AST

111.00

115.50

-4.05

2.78

-1.27

3.8

0.73

-0.33

111.0

Bilirubin, Direct

40.50

37.56

7.26

3.03

10.29

11.1

0.27

0.92

37.56

Bilirubin, Total

84.00

73.11

12.96

-0.13

12.84

2.4

-0.05

5.41

73.11

Calcium

2.220

2.140

3.60

-3.12

0.48

1.9

-1.64

0.25

2.22

Choloesterol

3.990

3.960

0.75

-1.85

-1.09

2.3

-0.82

-0.49

3.99

Creatine Kinase

350.00

334.87

4.32

-1.44

2.88

7.6

-0.19

0.38

334.87

Creatinine

331.00

322.34

2.62

0.41

3.03

3.8

0.11

0.81

322.34

Iron

38.50

39.09

-1.53

5.51

3.97

7.7

0.72

0.52

38.5

GGT

93.60

91.94

1.77

-1.97

-0.20

6.3

-0.32

-0.03

93.6

Glucose

11.30

11.56

-2.30

-0.18

-2.48

1.6

-0.11

-1.57

11.56

Lactate

2.99

3.04

-1.67

0.00

-1.67

7.6

0.00

-0.22

3.04

LDH

237.00

239.56

-1.08

1.50

0.42

2.9

0.53

0.15

237.0

Lipase

72.50

74.50

-2.76

-3.46

-6.22

7.3

-0.48

-0.86

74.5

Magnesium

1.050

1.110

-5.71

2.32

-3.39

6.3

0.37

-0.54

1.11

Phosphorus

1.490

1.460

2.01

-0.51

1.50

1.1

-0.47

1.40

1.46

Salicylate

145.00

140.40

3.17

0.00

3.17

6.3

0.00

0.51

140.4

Triglycerides

1.540

1.550

-0.65

8.96

8.31

7.0

1.28

1.19

1.54

Total Protein

50.60

49.67

1.84

-0.76

1.08

2.5

-0.30

0.43

49.67

Uric Acid

290.00

293.99

-1.38

4.67

3.29

3.0

1.57

1.11

290.0

Urea

15.10

15.28

-1.19

1.03

-0.17

3.9

0.26

-0.04

15.1

Application of the above concepts based on six sigma principles is a “project-in-progress” in Winnipeg, Manitoba, Canada. It has already served to bring attention and interest in six sigma methodology in our region. Expectantly, it is a system to objectively assess how our automated analytic tests are doing, to determine where improvement is still needed and to measure where progress has been made. As such, it is fundamental to achieving six sigma quality performance.

References

1. www.westgard.com/essay36.htm

2. www.westgard.com/biodatabase1.htm

3. Westgard JO. Six Sigma Quality Design & Control, Westgard QC, Inc., 2001, Appendix 2, page 276