Establishing QC Acceptability Criteria for CLIA

It's no revelation to say that there is a lack of clear and specific directions about quality control in government and literature. In up to 30% of laboratories, this has led to ill-defined QC practice. This QC application provides more specific directions on how to select appropriate "QC acceptability criteria" and a Total QC strategy that will satisfy CLIA requirements.

In discussing "the myth of medical decision limits," it was noted that the lack of clear and specific directions in the literature has led to an ill-defined QC practice that apparently is used in up to 30% of laboratories. In previous QC planning applications, we've illustrated a well-defined, quantitative QC planning process that makes use of power function graphs, critical error graphs, and OPSpecs charts, as well as making use of analytical total error requirements and clinical decision interval requirements. Our purpose this month is to provide more specific directions on how to select appropriate "QC acceptability criteria" and a Total QC strategy that will satisfy CLIA requirements. These directions are consistent with our earlier illustrations of the QC planning process, but should clarify how to use different resources, such as the power function graphs available in the scientific literature, the OPSpecs charts available in printed form, computer support for manual selection, and computer support for automatic selection.

Page references for CLIA QC guidelines
CLIA standard for QC procedures
CLIA acceptability criteria and Total QC strategy
Detailed procedures for selecting QC acceptability criteria
From power function graphs in the literature
From the OPSpecs Manual
Manually using the QC Validator 1.1 program
Automatically using the the QC Validator 2.0 program

Page references for CLIA QC requirements

CLIA uses the term quality control to refer to 'those standards required to monitor and control the quality of the analytical testing process to assure the accuracy and reliability of the patient test result." [Fed Reg 1992(Feb 28);57(No 40);7114-51].

Subpart K defines general quality control standards for tests of moderate or high complexity and includes standards for:

Facilities [493.1204, page 7163];
Test methods, equipment, instrumentation, reagents, materials, and supplies [493.1205, page 71631]
Procedure manual [493.1211, page 7164];
Establishment and verification of method performance specifications [493.1213, page 7164];
Equipment maintenance and function checks [493.1215, page 7164-5];
Calibration and calibration verification procedures [493.1217, page 7165];
Quality control procedures [493.1218, page 7166];
Remedial actions [493.1219, page 7166-7];
Quality control records [493.1221, page 7167], and
Special standards applicable for specific conditions [493.1223 to 493.1283, pages 7167-72].

CLIA standard for statistical QC procedures

A laboratory can satisfy CLIA requirements by following the manufacturer's instructions for quality control that have been cleared, provided these instructions require that control materials be tested in the same manner as patient specimens, statistical parameters (i.e., mean and standard deviation) be determined for each lot of material through repetitive testing, and patient test results not be reported unless control results meet the laboratory's criteria for acceptability.

For any method not cleared as meeting CLIA QC requirements, "the laboratory must evaluate instrument and reagent stability and operator variance in determining the number, type, and frequency of testing calibration or control materials and establish criteria for acceptability used to monitor test performance during a run of patient specimens." [493.1218, page 7166]

QC acceptability criteria and Total QC strategy

In both situations, QC acceptability criteria must be established to monitor the daily operation of laboratory testing processes. Establishing QC acceptability criteria requires definition of the control rules to be used for evaluating QC data and the number of control measurements (N) to be evaluated. This can be accomplished using the QC planning process described earlier.

Since the purpose of QC is to assure the accuracy and reliability of patient test results, the selection of these QC acceptability criteria should assure the observed performance of the method satisfies the quality required for patient test results. CLIA also defines the minimum quality required for regulated analytes with its proficiency testing criteria for acceptable performance. Thus, the CLIA PT criteria can be used as the minimum quality requirements for selecting control rules and N's.

Ideally, the QC rules and N you select should provide 90% or greater detection of medically important errors and, at the same time, maintain a false rejection rate of 5% or less, preferably as low as 1-2%. In cases where 90% error detection can not be achieved, lower error detection may be acceptable if your measurement procedure is very stable and has few errors that need to be detected. In such cases, your Total QC strategy for analytical quality assurance requires, in addition to statistical QC, careful attention to preventive maintenance, instrument function checks (e.g., background or baseline checks), measurement performance tests (e.g., calibration verification, working range verification, comparison of test results between different analytical systems), and possibly patient data QC.

In selecting candidate QC procedures, you can improve error detection by increasing N, narrowing the control limits for single rule procedures, or using multi-rule QC procedures. You can reduce false rejections by decreasing N, widening control limits, and decreasing the number of rules in multi-rule procedures. Unfortunately, the factors that increase error detection also increase false rejections, and vice versa, thus it may be necessary in some situations to allow a higher rate of false rejections in order to achieve the desired error detection.

Detailed procedures for selecting QC acceptability criteria

Detailed procedures are presented for selecting QC acceptability criteria using (a) power function graphs from the literature, (b) OPSpecs charts from the OPSpecs Manual, (c) manual selection from OPSpecs charts prepared by the QC Validator program (version 1), and (d) automatic selection by the QC Validator program (version 2).

Selecting QC acceptability criteria using power function graphs from the literature

1. Define the quality requirement for the test (TEa).

For the test of interest, use the CLIA proficiency testing criterion for acceptable performance as the minimum quality requirement that must be achieved. See the CLIA table on this website for specific values. Convert to percent when necessary by use of an appropriate medical decision concentration or level.

2. Evaluate the imprecision (smeas) and inaccuracy (biasmeas) of the method.

Estimate the method's performance from method evaluation experiments, internal QC data, and/or proficiency testing survey results, whichever are most useful. Express as a percent of the critical medical decision level.

3. Calculate the critical-size systematic error (SEcrit) to be detected by QC.

Using the percent values for the quality requirement of the test and the imprecision and inaccuracy of the method, calculate the critical systematic error from the following equation: SE_crit = [ (TE_a - bias_meas)/1.65s_meas] - 1.65

4. Look up power function graphs for the QC procedures of interest.

Possible sources include textbooks in clinical chemistry, QC monographs, and papers in the scientific literature.

Westgard JO, Groth T. Power functions for statistical control rules. Clin Chem 1979;25:863-869.
Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the Quality and Productivity of Analytical Processes. Washington, DC: AACC Press, 1984, pp 195- 217.
Cembrowski GS, Carey RN. Laboratory Quality Management. Chicago: ASCP Press, 1989.
Westgard JO. OPSpecs Manual - Expanded Edition. Ogunquit, ME:Westgard QC, pp 2-15 to 2-21.

5. Assess the probabilities for error detection and false rejection.

Determine the probabilities for false rejection (Pfr) from the y-intercepts of the power curves. Determine the probabilities for error detection (P_ed) by drawing the critical systematic error on the power function graph, identifying the points of intersection with the power curves, and reading the corresponding probabilities for rejection from the y-axis.

6. Select control rules and the total number of control measurements (N).

Look for a control procedure that provides a P_ed of 0.90 or 90% detection of the critical systematic error and maintains a low P_fr, 0.05 or 5% or lower. Keep N as low as possible. Use single rules when possible.

6a. If 90% error detection cannot be achieved, 50% may be acceptable for an analytical system that is stable and only occasionally has a problem. Increase N to improve error detection. Use multirule control procedures to look back at previous control data and improve detection of errors that persist from one run to the next.

6b. If 50% error detection cannot be achieved, allow the false rejection rate to increase to the 5-10% range. Use multirule control procedures with Ns as high as 6 per run.

7. Adopt a Total Quality Control strategy.

Based on the expected error detection of the QC procedure, select a TQC strategy that provides an appropriate balance between statistical and non-statistical components of the QC system.

7a. If the QC procedure provides 90% error detection, the TQC strategy can depend primarily on statistical QC to test the total analytical process. Perform the minimum other QC checks as required by regulations, manufacturer's directions, and good laboratory practices.

7b. If the QC procedure provides 50% error detection or better, method stability is critical and the TQC strategy should place increased emphasis on preventive maintenance, instrument function checks, and performance validation tests to identify problems and prevent errors from occurring. Efforts to improve method performance should attempt to eliminate any observed bias and reduce imprecision.

7c. If the QC procedure provides less than 50%, the TQC strategy should place increased emphasis on preventive maintenance, instrument function checks, performance validation tests, patient data QC, and operator training and experience to identify problems and prevent errors from occurring. Efforts to improve method performance may consider replacement of this method by a more highly automated with better analytical performance.

8. Document the QC procedure and the TQC strategy.

Include the quality requirement for the test, the information and source of method performance data, the source of power function graphs, the calculated critical systematic error, the expected probabilities for error detection and false rejection, and the components and schedule for the TQC strategy. Identify methods that are problematic and in need of improvement. Set priorities for improvement of existing methods and the development and/or acquistion of new methods.

Selecting QC acceptability criteria using the OPSpecs Manual

1. Define the quality requirement for the test (TE_a).

2. Evaluate the imprecision (s_meas) and inaccuracy (bias_meas) of the method.

3. Lookup the OPSpecs chart.

Using the OPSpecs Manual, find the OPSpecs charts for the quality requirement defined and the number of control materials to be analyzed. For 2 control materials, use the charts for N=2 and N=4. For 3 control materials, use the charts for N=3 and N=6.

4. Plot your operating point.

Plot your observed imprecision (as x) and inaccuracy (as y) on the OPSpecs charts to locate your operating point. Begin with the lower N chart and 90% AQA, then go to the higher N and 90% AQA, then to the 50% AQA charts. Remember that lines passing below your operating point do not provide the desired error detection; the lines passing above your operating point do provide the desired error detection and are possible candidates for your QC acceptability criteria.

5. Assess the probabilities for error detection and false rejection.

6. Select control rules and the total number of control measurements (N).

6a. If 90% error detection cannot be achieved, 50% may be acceptable for an analytical system that is stable and only occasionally has a problem. Increase N to improve error detection. Use multirule control procedures to look back at previous control data and improve detection of errors that persist from one run to the next.

6b. If 50% error detection cannot be achieved, allow the false rejection rate to increase to the 5-10% range. Use multirule control procedures with Ns as high as 6 per run.

7. Adopt a Total Quality Control strategy.

Based on the expected error detection of the QC procedure, select a TQC strategy that provides an appropriate balance between statistical and non-statistical components of the QC system.

7a. If the QC procedure is selected from a 90% AQA chart, the TQC strategy can depend primarily on statistical QC to test the total analytical process. Perform the minimum other QC checks as required by regulations, manufacturer's directions, and good laboratory practices.

7b. If the QC procedure is selected from a 50% AQA chart, method stability is critical and the TQC strategy should place increased emphasis on preventive maintenance, instrument function checks, and performance validation tests to identify problems and prevent errors from occurring. Efforts to improve method performance should attempt to eliminate any observed bias and reduce imprecision.

7c. If the QC procedure is below the operating limits on a 50% AQA chart, the TQC strategy should place increased emphasis on preventive maintenance, instrument function checks, performance validation tests, patient data QC, and operator training and experience to identify problems and prevent errors from occurring. Efforts to improve method performance may consider replacement of this method by a more highly automated with better analytical performance.

8. Document the QC procedure and the TQC strategy.

Manual selection of QC acceptability criteria using the QC Validator program (version 1)

1. Start the QC Validator program.

Double-click on the Validator icon in the Windows program manager. If you have used QC Validator before, you are an existing user and only need to enter your password to sign-on. If you are a new user, enter the information requested on the Sign-on Window and establish a password for future sign-ons.

2. Enter the information requestion on the Parameters Window.

This includes information about the analytical system, the test name, the test units, the medical decision concentration or level, as well as the imprecision and inaccuracy observed for your method and the quality requirement you wish to achieve.

2a. Enter the imprecision and inaccuracy observed in units of percent for the decision level of interest.

2b. Enter the CLIA proficiency testing criterion for acceptable performance in units of percent. Look up the CLIA requirement in the HELP facility of the program or in tables at the end of the program manual.

3. Select candidate QC procedures that are of interest.

Use the "common N" buttons for common single and multirule procedures with the N shown on the button. Select QC procedures "from table" to access the entire list of available rules and Ns.

4. Obtain OPSpecs chart.

Using the Options menu, select OPSpecs charts to obtain the 90% AQA chart for the selected candidate QC procedures. Remember that lines passing below your operating point do not provide the desired error detection; the lines passing above your operating point do provide the desired error detection and are possible candidates for your QC acceptability criteria.

5. Assess the probabilities for error detection and false rejection.

Inspect the OPSpecs charts and identify control rules and Ns corresponding to those lines above your operating point. Candidate QC procedures selected from the 90% AQA chart provide at least 90% error detection; lookup Pfr in the key area at the right of the chart. Access 50% AQA and 25% AQA charts as needed by clicking the button on the screen. Try higher Ns by returning to the parameters window and selecting new candidate QC procedures.

6. Select control rules and the total number of control measurements (N).

Look for a control procedure that provides a P_ed of 0.90 or 90% detection of the critical systematic error and maintains a low P_fr, 0.05 or 5% or lower. Identify your selection by clicking on the operating line in the chart area or in the key area of the chart. Keep N as low as possible. Use single- rules when possible.

6b. If 50% error detection cannot be achieved, allow the false rejection rate to increase to the 5-10% range. Use multirule control procedures with Ns as high as 6 per run.

7. Adopt a Total Quality Control strategy.

Based on the expected error detection of the QC procedure, select a TQC strategy that provides an appropriate balance between statistical and non-statistical components of the QC system.

7a. If the QC procedure is selected from a 90%AQA chart, the TQC strategy can depend primarily on statistical QC to test the total analytical process. Perform the minimum other QC checks as required by regulations, manufacturer's directions, and good laboratory practices.

8. Document the QC procedure and the TQC strategy.

Use the QC Validator Report function to document the QC acceptability criteria, input parameters, calculated parameters, and the appropriate OPSpecs charts, critical-error graphs, or power function graphs. Document the components and schedule for the TQC strategy. Identify methods that are problematic and in need of improvement. Set priorities for improvement of existing methods and the development and/or acquistion of new methods.

Automatic selection of QC acceptability criteria using the QC Validator program (version 2)

1. Start the QC Validator program.

2. Enter the information requestion on the Parameters Window.

2a. Enter the imprecision and inaccuracy observed in units of percent for the decision level of interest.

2b. Set the method instability input to an appropriate value (10%) based on the expected frequency of problems for the method.

2c. Enter the CLIA proficiency testing criterion for acceptable performance in units of percent. Look up the CLIA requirement in the HELP facility of the program or in tables at the end of the program manual.

3. Decide whether to use 2 or 3 control materials.

Most highly automated methods can be adequately controlled with 2 materials. Common practice often calls for 3 materials with blood gas, hemology, coagulation, and immunoassay applications.

4. Initiate automatic selection by clicking the 2 or 3 control materials button.

Default selection criteria will consider single-rule and multirule procedures with total Ns of 2 and 4 for 2 materials and total Ns of 3 and 6 for 3 materials. The program will first attempt to find a QC procedure that provides 90% AQA (or error detection) and will make a second pass looking for 50% AQA if the method instability parameter has been set to <2%. If method instability is left as "off", then the automatic selection process will make only a single search looking for 90% AQA.

5. Review the OPSpecs chart showing the selected QC acceptability criteria.

If the selection is displayed on a 90% AQA chart, then at least 90% error detection will be achieved. If the selection is displayed on a 50% AQA chart, then at least 50% error detection will be achieved. If 50% AQA cannot be achieved, the program will default to manual selection from a table of "maximum QC procedures" or allow you to make a manual selection from the list of available QC procedures. Read the expected false rejection probability in the key area of the chart.

6. Approve the automatic selection or make a manual selection.

Manually click on any line on the OPSpecs chart to change the selection. Use the Options menu and the QC selection criteria option to review and modify the selection criteria to change the automatic selection process.

7. Adopt a Total Quality Control strategy.

Based on the expected error detection of the QC procedure, select a TQC strategy that provides an appropriate balance between statistical and non-statistical components of the QC system.

7a. If the QC procedure is selected from a 90%AQA chart, the TQC strategy can depend primarily on statistical QC to test the total analytical process. Perform the minimum other QC checks as required by regulations, manufacturer's directions, and good laboratory practices.

Tools, Technologies and Training for Healthcare Laboratories

Quality Standards