Tools, Technologies and Training for Healthcare Laboratories

Part III: The Quality of Cholesterol Testing

December 2004

Quality assessment is the major focus of the Final CLIA Rule [1], particularly the quality assessment of pre-analytic and post-analytic processes. For analytical processes, the Final Rule has increased the requirements for method validation, reduced the requirements for QC to generally 2 controls per day, and proposed further reductions in QC for methods having built-in internal controls [2]. CMS (Centers for Medicare and Medicaid Services) provides the guidelines for reduced QC, or “Equivalent QC”, in the State Operations Manual, SOM [3]. EQC allows the minimum requirement of 2 controls per day to be further reduced to weekly or even monthly QC for methods with built-in controls.

Such reductions seem to be the wrong direction, considering the recent laboratory QC problems at Maryland General Hospital [4]. Laboratories today may not be doing adequate QC, as documented by CMS’s own survey results that show 5-10% of laboratories today still have QC deficiencies [4]. However, CMS seems to believe that current methods and technology no longer require rigorous QC, though no data exists to support this belief.

We recognize that "faith-based" initiatives are very popular with the current administration, but we don’t think the American public is ready to accept "faith-based QC" for laboratory testing. Instead, we advocate the application of Six Sigma Quality Management to provide an objective assessment of the quality of laboratory testing and the amount of QC that is needed [5]. This assessment can be readily carried out using proficiency testing results [6], which are required by CLIA and must be reported to CMS. Our premise is that the amount of QC depends on the quality observed for the testing process [7]. Minimum QC applies when 6 Sigma performance is achieved; more QC is required for lower sigmas. When less than 3 Sigma performance is observed, the laboratory methods aren’t actually suitable for routine testing.

Materials and Methods

Sigma metrics are estimated for cholesterol testing based on proficiency testing data collected during 2004.

  • National requirement for the quality of cholesterol is defined as an allowable total error (TEa) of 10.0%, which is the CLIA criterion for acceptable performance in proficiency testing (PT).
  • Survey specimens were selected near the medically important decision concentration of 200 mg/dL.
  • PT data comes from 2004 surveys performed by the American Academy of Family Physicians (AAFP), Medical Laboratory Evaluation (MLE), American Association of Bioanalysts (AAB), American Proficiency Institute (API), College of American Pathologists (CAP), and New York State (NY).
  • National Test Quality (NTQ) observed for a single proficiency testing sample is estimated from the CLIA total allowable total error (TEa) divided by the group SD or CV, i.e., Sigma = TEa/CV. The average NTQ observed for multiple surveys is weighted for the number of laboratories participating in the survey.
  • Local Method Quality (LMQ) for a single proficiency testing sample is a weighted average of the Sigmas determined for each method subgroup without accounting for method bias, i.e., Sigma = TEa/CVmethsubgroup. The average LMQ observed for multiple surveys is weighted for the number of laboratories participating in each survey.
  • National Method Quality (NMQ) observed for a single proficiency testing sample is a weighted average of the Sigmas determined for each method subgroup taking bias into account, i.e., Sigma = (TEa – biasmethsubgroup)/CVmethsubgroup. The average NMQ observed for multiple surveys is weighted for the number of laboratories patricipating in each survey.

Further details on the methodology are discussed in an earlier essay.

Results

Estimates of quality. Table 1 shows the estimates of sigma performance for cholesterol tests from 5 different PT survey programs, which are identified in Column 1. Column 2 shows the number of labs surveyed by each program, with a total of 9,258 laboratories represented by all these programs. AAFP and MLE each represent a few hundred labs, whereas AAB, API, and CAP represent from one thousand to four thousand labs.

Table 1. Summary of cholesterol quality from 5 national PT survey programs

Sigma Quality Performance Metrics

PT Program

Labs

Group Mean

National Test Q

Local Method Q

National Method Q

Datasheet

AAFP 2004A

296

201.0

2.01

2.54

2.01

1

MLE M2

577

224.4

2.27

2.99

2.38

2

AAB 2nd 2004

1498

223.0

2.37

3.51

2.68

3

API 2nd 2004

2647

221.3

2.28

3.19

2.37

4

CAP LP-01 2004

4240

198.7

3.57

4.19

3.71

5

Weighted average

9258

210.8

2.88

3.67

3.02

NY Benchmark

371

208.1

4.09

4.83

4.48

6


Column 3 shows the group means range from 198 to 223, with an average of about 211 mg/dL, thus these selected PT specimens have concentrations that are close to the 200 mg/dL decision level where medical interpretation is important, according to NCEP guidelines.

Column 4, National Test Quality, is estimated as 2.01 Sigma from the AAFP survey data, 2.27 Sigma from the MLE survey, 2.37 from AAB, 2.28 from API, and 3.57 from CAP, which gives a weighted average of 2.88 Sigma performance. As a benchmark for comparison, the NY State survey provides a Sigma of 4.09, which is expected to represent the quality of laboratory testing in the most demanding regulatory program in the US.

Note that the numbers in Column 6, National Method Quality, match quite closely with those in Column 4. This agreement provides some indication that the different calculations lead to consistent results.

Column 5, Local Method Quality, provides the highest or best sigmas because these estimates disregard any between-method biases.

Column 7 provides a reference to the datasheets that contain all the raw numbers.

Variability of estimates. Table 2 provides a more detailed assessment for CAP results for 2 testing events and a total of 10 specimens. Estimates of National Test Quality and National Method Quality are again in good agreement, except for specimen LP-03, where the difference of 2.78 to 3.22 (or 0.46 Sigma) is the largest observed. The estimates of Local Method Quality are the most consistent, with an average of 4.16, a standard deviation of 0.12, and an observed range 3.98 to 4.39 (minimum to maximum), which falls far short of the 6-Sigma goal for World Class Quality (and minimum QC).

Table 2. Summary of Cholesterol Quality from 10 CAP 2004 Specimens

Sigma Quality Performance Metrics

PT Specimen

Number Labs

Group Mean

National Test Q

Local Method Q

National Method Q

Datasheet

LP-01

4240

198.7

3.57

4.19

3.71

7

LP-02

4181

153.9

3.13

3.98

3.37

8

LP-03

4194

241.3

2.78

4.11

3.22

9

LP-04

4191

252.4

3.70

4.12

3.78

10

LP-05

4178

252.7

3.70

4.28

3.91

11

LP-06

4292

177.1

3.57

4.17

3.79

12

LP-07

4274

199.2

3.85

4.23

3.98

13

LP-08

4262

156.6

3.57

4.12

3.73

14

LP-09

4268

244.0

3.03

4.04

3.35

15

LP-10

4248

177.0

3.70

4.39

4.04

16

Avg Evt 1



3.38

4.14

3.60

SD



0.41

0.11

0.29

Avg Evt 2



3.54

4.19

3.78

SD



0.31

0.13

0.27

Avg All



3.46

4.16

3.69

SD

0.35

0.12

0.28

MEDIAN

3.57

4.15

3.75

MINIMUM

2.78

3.98

3.22

MAXIMUM

3.85

4.39

4.04



Discussion

The Sigma metrics for National Test Quality and National Method Quality in smaller laboratories range from 2.01 to 2.68 Sigma, based on 4 different surveys that include 5,018 laboratories. These estimates for larger laboratories represented by CAP data are 3.57 and 3.71 based on 4,240 laboratories. This suggests that the quality of cholesterol testing does depend on the size of the laboratory and the sophistication of the methods and analysts involved. The NY State benchmark figures support this assessment, showing that Sigma performance in the range of 4 to 5 can be achieved by highly automated methods in large laboratories. The overall weighted averages for the 9,258 laboratories are 2.88 Sigma for NTQ and 3.02 Sigma for NMQ, as shown in the graph below.

The most optimistic estimates of quality, i.e., Local Method Quality, range from 2.54 to 3.19 in smaller laboratories (AAFP, MLE, AAB, API data) and 4.19 in larger laboratories (CAP data). The overall weighted average for the 9,258 laboratories is 3.67 Sigma for LMQ, as shown on the graph below. Note that even this most optimistic estimate of method performance falls far short of World Class Quality.

In all cases, these estimates of Test and Method Quality challenge the adequacy of current CLIA regulations for the minimum QC of 2 controls per day. Methods with 4 Sigma performance need to be controlled with 4 controls per run. Methods with 3 Sigma performance, or less, are simply cannot be controlled to achieve the national quality requirement specified by the CLIA PT criterion for cholesterol.

Conclusion

Strike one!

References

  1. US Centers for Medicare & Medicaid Services (CMS). Medicare, Medicaid, and CLIA Programs: Laboratory Requirements Relating to Quality Systems and Certain Personnel Qualifications. Final Rule. Fed Regist Jan 24 2003;16:3640-3714.
  2. Westgard JO, Ehrmeyer SS, Darcy TP. CLIA Final Rules for Quality Systems: Quality Assessment Issues and Answers. Westgard QC, Inc., Madison, WI, 2004
  3. Appendix C of SOM, Regulations and Interpretive Guidelines for Laboratories and Laboratory Services.
  4. Westgard JO, Westgard S. Maryland General Hospital. Part V. A few bad apples or the tip of the iceberg?
  5. Westgard JO, Westgard S. The Quality of Laboratory Testing. Part I. Myths vs metrics.
  6. Westgard JO, Westgard S. The Quality of Laboratory Testing. Part II. Touchstone Test Methodology.
  7. Westgard JO. Appropriate QC.

James O. Westgard, PhD, is a professor of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.