CLIA Final Rule:

Appropriate QC Procedures

December 2003

James O. Westgard, Ph.D.

In principle, the CLIA regulations require that laboratories perform the amount of QC that is appropriate for the methods in use in the laboratory. That’s the real meaning of section 493.1256 [1] and the following rules:

In practice, CLIA also provides an easy way out, which is to run two levels of control every twenty four hours. You can make a choice between being in compliance with the regulations or being in control of quality! If you choose to be in compliance, you don’t need to read any further. Just run two levels of control every 24 hours. If you choose to control quality, that means doing the right QC for the methods in your laboratory so you can detect immediate errors.

What QC is right for your methods?

The problem is to figure out the right QC based on the accuracy and precision of the method, the number of control materials to be tested, and the size of the errors that are medically important and need to be detected immediately.

This issue comes up again and again at meetings, workshops, and seminars. People often hope for a simple answer, a single rule or set of rules that is right for all tests and all methods. Unfortunately, there isn't any single QC procedure that is right for all methods, not even multirule QC using "Westgard Rules"! However, there is a right QC procedure for each of your methods. That QC procedure can be identified by following a QC selection process that considers the quality required for the test, the precision and accuracy observed for the method, and rejection characteristics of the particular control rules and number of control measurements used by the QC procedure. The answer is not a rule or set of rules, but a process that leads to the right rule or rules for an individual test and method.

In this discussion, we're going to simplify this selection process and break it down into several steps:

How do you characterize the performance of your method?

A simple approach is to calculate the Sigma metric for your method [2] following these steps:

Sigma = (TEa - bias)/SD

For cholesterol, for example, the CLIA criterion is an allowable total error of 10%. If your method has a CV of 2.0% and a bias of 1.0%, then the Sigma metric for your method is 4.5 [(10-1)/2]. This application is illustrated in the first figure, where the quality requirements are shown by the lines in red, method imprecision by the normal distribution curve, and method bias by the displacement of the mean of the distribution curve towards the right.

In interpreting the method Sigma metric, remember that the higher the Sigma value, the better the method performance. Established performance benchmarks are as follows:

For sources of the numbers in the Sigma calculation, the CLIA proficiency testing criteria can be found in the CLIA regulations or in summary tables, and is provided on this website. For figures for the method SD and bias, CLIA requires that you verify the manufacturer’s claims for precision and accuracy for any method introduced after April 24, 2003, thus there should be data on the method SD and bias available in your own laboratory from your initial replication (SD) and comparison experiments (bias vs comparative method), respectively, or from ongoing estimates from your routine QC data (SD) and periodic peer comparison of proficiency testing surveys (bias vs mean of group). Alternatively, as a starting point, you can begin with the manufacturer’s claims for precision and accuracy, which must be available to you before any method or system can be marketed and sold.

Examples for practice (answers can be found at end of discussion):

For a cholesterol method, what is the sigma metric if the method CV is 3.0% and bias is 1.0%?

For a cholesterol method, what is the sigma metric if the method CV is 2.0% and the bias is 2.0%?

For a glucose method, what is the sigma metric if the method CV is 2.0% and the bias is 0.0% at a concentration of 125 mg/dL?

For blood pH, what is the sigma metric if the method SD is 0.005 and the bias is 0.01 pH unit?

To demonstrate the usefulness of calculating method’s Sigma performance, I’ll skip to the end and show you the QC that is needed for methods having different sigma metrics:

Once you know the sigma metrics for your methods, the above guidelines will give you a pretty good idea of what QC is right for those methods. If you want to understand the rationale for these guidelines, keep reading.

What size errors are medically important?

All test results are in error to some extent due to the inherent random error caused by the stable imprecision of the method. If a method has an SD, then there will be some error in the test results! Inaccuracy can ideally be reduced to zero, but imprecision will always exist, thus test results will always have some analytical error. The real issue is the size of an error that becomes medically important and needs to be detected by the QC procedure. For automated systems, we are most concerned with the systematic errors that occur due to deterioration of instrument components and changes in reagents, operators, and operating conditions.

The second figure shows how to define the systematic error that is medically important. We call this the "critical SE" and have used the abbreviation "delta SE critical" (SEcrit) in many discussions on this website and in our training materials on QC design and planning. To calculate the medically important systematic error, we limit the "risk" for producing a bad test result to 5%, which means that the area in tail is defined a z-value of 1.65s, thus providing the following equation for the medically important error that needs to be detected by the QC procedure;

SEcrit = [(TEa - bias)/s] - 1.65s

We utilize SEcrit as the Sigma Design metric for quality control [3]. It is a Sigma metric because it describes the medically important systematic error as a multiple of the method SD. For example, a critical SE of 3.0s means that it is important to detect a systematic shift equivalent to 3.0 times the method SD. If the method SD is 2.0 mg/dL, then the critical shift amounts to 6.0 mg/dL. If the method CV is 2.0%, the critical shift is 6.0%, which can be converted to concentration units at a critical decision concentration.

For our cholesterol example, where TEa is 10%, bias is 1.0%, and SD is 2.0%, the medically important systematic error is 2.85s, i.e., a systematic shift equivalent to 2.85 times the standard deviation of the method needs to be detected by the QC procedure to assure the desired quality will be produced for a cholesterol test.

Note that the equation for the QC design metric could also be written as follows:

SEcrit = Method Sigma - 1.65s

That means that the requirement for the QC procedure is directly related to the Sigma performance of the method, i.e., the higher the method Sigma, the larger the size of the systematic error that needs to be detected by the QC procedure. The larger the error, the easier it is for the QC procedure to detect the error!

What size errors can be detected by a QC procedure?

While it's easy to calculate the sigma metric to describe the performance of a method in your own laboratory and also to calculate the medically important systematic error (QC Design metric) that must be detected by the QC procedure, most laboratories don't know the error detection capabilities of their QC procedures. Fortunately this information has been available in the scientific literature for many years [4].

The performance of a QC procedure can be described by two characteristics, as noted earlier in the discussion of "Equivalent QC procedures" on this website:

Ideally, Ped should be high, near 1.00 to provide a 100% chance of detecting a medically important analytical error. Ideally, Pfr should be low, near 0.00 to provide a 0.0% chance of false rejections that would otherwise waste time and effort and slow the reporting of patient test results.

These probabilities can be described graphically using a Power Function graph [5], as shown in the figure below. Note that there are 8 power curves here, each corresponding to specific control rules and a specific number of control measurements, as shown in the key at the right side. We generally set the desired error detection at 0.90 (y-axis, probability for rejection), as shown by the dashed horizontal line. The capability of the QC procedure to achieve that performance depends on the size of the error to be detected (x-axis, systematic error). The vertical line shows the size of the medically important systematic error for our cholesterol example (2.85s). Note that only the top 3 QC procedure on the list in the key provide the desired error detection. Of these 3, the 2nd and 3rd down the list require 4 control measurements and are therefore less costly than the 1st on the list, which requires 6 control measurements. Note also that the probability for false rejection will be lower for the N=4 QC procedures.

What QC is appropriate for methods with different Sigma metrics?

Since the medically important systematic error is directly dependent on the Sigma metric of the method, it follows that the QC procedure that is appropriate must also be related to the sigma metric of the method. A quantitative description of the relationship is provided in the figure below, which shows a Sigma scale imposed on the power function graph and vertical lines corresponding, from left to right, to 3, 4, 5, and 6 Sigma (slightly off-scale at the right).

Given a method with 3 Sigma performance (leftmost vertical line), a laboratory needs to do all the QC it can afford. Even a multirule QC procedure with 6 controls per run will have difficulty providing the desired error detection.

On the other hand, a method having 6 Sigma performance (rightmost vertical line) can be QC'd by any of these rules, certainly 13s with N=2 and even 13s with N=1. Given that CLIA requires a minimum of 2 levels of controls, a laboratory could even use a 13.5s rule with N=2 to effectively reduce false rejections to zero and still maintain the desired error detection.

For a method with 5 Sigma performance, 2 levels with either a 13s/22s/R4s multirule or a 12.5s single rule would be appropriate. For a method with 4 Sigma performance, it will be necessary to increase N to 4, which means analyzing each of the two levels of controls twice to get the total 4 measurements that are needed.

In summary, when your method Sigma is 6 or greater, you can do QC anyway you want, just be sure to keep the false rejections low by using wide control limits - at least 3s. When your method Sigma is 5 or so, use N=2 or 3 with 2.5s or 3.0s control limits. When your method Sigma is 4 or so, increase N to 4 or 6 and use either the 12.5s single rule or a 13s/22s/R4s/41s multirule procedure. With method Sigmas below 4.0, run all the control you can afford. In addition, increase the frequency of instrument function checks, performance validation checks, and preventive maintenance. With method Sigmas below 3.0, look for a new and better method.

For a more quantitative QC selection process that is supported by computer tools, see the recent publication on Internal Quality Control Planning and Implementation Strategies [6] which is available on the Internet.

How can you document the error detection of your QC procedures?

When the inspector comes calling on your laboratory, you can demonstrate the error detection of your QC procedures with the calculations and graphs discussed here. This documentation is readily available with the EZ Rules® computer program, which provides the Sigma metric calculations and graphics shown in the final figure. This program also provides an Automatic QC Selection function that will search for the optimal control rules and number of control measurements based on your entries of the quality requirement, observed imprecision, and observed inaccuracy. The program also allows you to utilize biologic goals and medically important changes (clinical decision intervals) as quality requirements. Thus EZ Rules® makes it easy to select appropriate QC procedures and document their appropriateness for use with the methods in your laboratory.

Click here to view a PDF copy of a “QC Validation Report” to demonstrate the documentation that is provided by the EZ Rules computer program.

Answers to example calculations for method Sigma metrics

(a) Cholesterol with method CV of 3.0% and bias of 1.0% is 3.0 Sigma [(10-1)/3];

(b) Cholesterol with method CV of 2.0% and bias of 2.0% is 4.0 Sigma [(10-2)/2];

(c) Glucose with method CV of 2.0% and bias of 0.0% is 5.0 Sigma [(10-0)/2]; and

(d) pH with method SD of 0.005 pH unit and bias of 0.01 pH unit is 6.0 Sigma [(0.04-0.01)/0.005].

And the appropriate QC procedures are:

(a) All the QC you can afford;

(b) Use 2.5s limits for a single rule or a multirule procedure having 13s/22s/R4s/41s with a total N of at least 4, which means making duplicate measurements on each of two different levels of materials,

(c) Use and N of 2 with 2.5s or 3.0s for a single rule procedure, or a simple multirule having 13s/22s/R4s rules.

(d) Do QC anyway you want, just be sure to keep the N low and the control limits wide to minimize cost.

References

  1. CLIA Final Rules. U.S. Department of Health and Human Services. Medicare, Medicaid and CLIA Programs: Laboratory requirements relating to quality systems and certain personnel qualifications. Final rule. Fed Regist 2003;68:3640-714 (available at http://www.phppo.cdc.gov/clia/pdf/CMS-2226-F.pdf)
  2. Crolla LJ, Westgard JO. Evaluation of rule-based autoverification protocols. CLMR 2003;17:268-272.
  3. Westgard JO. Six Sigma Quality Design and Control: Desirable precision and requisite QC for laboratory measurement processes. Madison, WI: Westgard QC, Inc., 2001.
  4. Westgard JO, Groth T, Aronsson T, Falk H, deVerdier C-H. Performance characteristics of rules for internal quality control: Probabilities for false rejection and error detection. Clin Chem 1977;23:1857-1867.
  5. Westgard JO, Groth T. Power function graphs for statistical control rules. Clin Chem 1979;25:863-869.
  6. Westgard JO. Internal quality control: planning and implementation. Ann Clin Biochem 2003;40:593-611.

See the Entire CLIA Final Rule Series: