Tools, Technologies and Training for Healthcare Laboratories

Evolution of QC Design Tools

June 2006

There are a variety of tools and programs available to those who want to implement Sigma metrics and QC Design in their laboratory. Some are free, some are not, and some new tools are coming soon...

Banner green Belt 33

June 2006

I’ve recently concluded my course on Analytical Quality Management for the senior CLS class at UW-Madison. The last section deals with QC Design and Planning and I thought the concluding discussion of available tools might also be of interest to the laboratory community. There also is a question on the advantages and disadvantages (limitations) of the different tools on the class’s final exam, so I need to prepare an answer anyway. Here it is!

There are 3 different tools currently available – Sigma-metrics graph, Normalized OPSpecs charts, and the EZ Rules 3 computer program. There is one new tool on the horizon – Bio-Rad’s Westgard Advisor program.

[Full disclosure: we're talking about products that we sell. Westgard QC, Inc. is a for-profit company and generates revenue from selling these products. Bio-Rad Laboratories purchased several patents and software rights from Westgard QC, Inc., so there is also revenue generated by sales of Westgard Advisor.]

Evolution of QC Design Tools

The development of QC design and planning tools goes back to 1976 when I was on sabbatical leave at Uppsala University in Sweden – that’s 30 years ago! As I’ve often mentioned, that was the beginning of my life’s work on quality control. The rest is history and is summarized in the following figure, which shows the development and evolution of quality planning tools since that time.

ess94f1

In Uppsala, we began to study the rejection characteristics of different QC procedures, which led to the development of a computer simulation program that would generate data sets having specified amounts of analytical error. These data sets were then tested with different control rules to determine which rules and numbers of control measurements would provide detection of the error conditions. By testing thousands of data sets, we could estimate the probability or chance of rejection for many different QC procedures, which was the focus of the first publication about QC performance characteristics [1]. This paper also described the merits of multirule QC procedures, which we later documented in much greater detail in the example application that has become known as “Westgard Rules.” [2]

We developed an interactive QC simulator program to allow a user to set all the parameters of interest in studying a particular QC application [3]. I needed that program when I returned to Wisconsin in order to carry out further studies. In 1980, we published the power function graphs for most of the QC procedures in common use [4]. This was followed in 1986 by guidelines for the cost-effective operation of analytical testing processes [5].

In the mid-80s, I become involved with Total Quality Management, which added a focus on customers and their requirements for quality. Up till this time, we were optimizing QC procedures based on their comparative probabilities for error detection and false rejection, but without regard for the specific size of errors that were medically important for the particular test. To take that into account, we began to use “critical-error graphs” [6], which were the precursor of today’s “Sigma-metrics QC Planning Tool.”

This focus on quality requirements led to the development of “quality-planning models” and the “chart of operating specifications” or “OPSpecs Tool.” When we published the use and applications of OPSpecs charts [7,8], we were challenged by a reviewer to make this stool available via a computer program in order to facilitate the use by other laboratory scientists. That led to the formation of Westgard QC as the mechanism to develop computer tools and support education and training in laboratory quality management. The first program was called “QC Validator”, which evolved later into Validator 1.1 with automatic QC selection. Then came the EZ Rules program to provide a more user friendly and supportive interface, followed by the integration of the automatic rule selection process into a QC charting program called EZ Runs. EZ Rules has now evolved to version 3, or EZ Rules 3, which supports multi-stage designs, including the design of “average of normals” patient data QC.

Most recently, Bio-Rad Laboratories has acquired the intellectual property rights to the automatic QC selection process and is introducing a “Westgard Advisor” program to provide guidance and recommendations to laboratories on how to optimize their QC procedures. By taking advantage of their peer-comparison data, estimates of method performance can be readily available and be applied for QC planning.

1. Sigma-metrics QC Design Tool

This tool is a power function graph [4] that shows the probability for rejection vs the size of the error for different QC rules and numbers of control measurements. The key to this tool is the critical systematic-error (?SEcrit) that needs to be detected by the QC procedure. The size of this error depends on the quality required for the test and the precision and accuracy observed for the measurement procedure, as follows [5,9]:

(?SEcrit = [(TEa – Biasobs)/SDobs] – 1.65

where TEa is the allowable total error,

Biasobs is the inaccuracy of the measurement procedure,

SDobs is the imprecision of the measurement procedure, and

all terms must be in the same units, either concentration or %.

In 1990, we demonstrated the use of this tool for application with a multitest chemistry analyzer [6]. What has changed now is the addition of a sigma-scale which makes the tool a little easier to use. Given that the expression [(TEa – Biasobs)/SDobs] can also be called a sigma-metric, which is more easily understood in light of current interests in Six Sigma Quality Management, the critical-error graph has been re-scaled to show the sigma-metric [10,11]:

Sigma-metric = [(TEa - Biasobs)/SDobs] = ?SEcrit + 1.65

The following figure shows the Sigma-metrics graph, where the critical systematic error is shown on the x-axis at the bottom of the graph, the sigma-scale is shown at the top of the graph, and the probability for error detection is shown on the y-axis.

ess94f2

Procedure [12]

  • Define the quality requirement in terms of TEa
  • Obtain estimates of precision and bias of measurement procedure
  • Calculate Sigma-metric
  • Locate calculated sigma-value on Sigma-metrics graph
  • Draw vertical line to intersect power curves
  • Select QC procedure that provides Ped of 0.90 or 90% error detection

Advantages

  • Easy to calculate the sigma-metric;
  • Easy to teach to laboratory scientists;
  • Illustrative applications provided in CLSI C24-A3 [13];
  • Provides “rules of thumb” that are easy to apply:
  • If sigma-metric 6 or greater, any QC will do;
  • If sigma-metric 5 or greater, N=2 okay;
  • If sigma-metric 4, need N=4;
  • If sigma-metric <4, need multirule QC
  • If sigma-metric <3, need new method!

Disadvantages

  • Works only with analytical quality requirement of form TEa;
  • Provides a limited choices of rules and Ns;
  • Must document application manually and separately.

2. Normalized Charts of Operating Specifications

The OPSpecs Chart is a graphic tool that shows the relationship between the quality requirement for the test, the precision and accuracy observed for a method, and the rejection characteristics for different control rules and numbers of control measurements [14]. Combining all that information on a single graph makes this tool complicated and difficult to understand, yet it is remarkably easy to use. You plot a point that represents the precision and accuracy observed of your method, then select a line above that point and identify the control rules and number of measurements for the QC procedure represented by that line.

This tool makes use of the equation for the critical systematic error, rearranged as follows:

?SEcrit = [(TEa – Biasobs)/SDobs] – 1.65

?SEcrit + 1.65 = (TEa - Biasobs)/SDobs

(?SEcrit + 1.65)*SDobs = (TEa - Biasobs)

 

Biasobs + (?SEcrit + 1.65)*SDobs = TEa

 

Biasobs = TEa - (?SEcrit + 1.65)*SDobs

Note that this last equation is of the form Y=a + bX, which represents a straight line whose slope depends on ?SEcrit, which is represented by the size of the critical error that can be detected with a specified probability, e.g., 0.90 or 90% detection, represented on the chart as AQA(SE) for “Analytical Quality Assurance for Systematic Error [AQA(SE)]. (This value is obtained from the power curve that represented the control rules and number of control measurements by specifying the desired Ped on the y-axis, identifying the point of intersection with the power curve, then identifying the size of the error from the x-axis of a power function graph [15].)

In the initial applications of OPSpecs charts, it was recommended that the chart be prepared for the quality requirement of interest. Because of the wide range of quality requirements, this led to the preparation of a manual of OPSpecs charts that covered all the values included in the CLIA criteria for acceptable performance [16]. Later planning guidelines [17] recommend the use of “normalized” OPSpecs charts, where only a handful of charts are needed when the observed imprecision and bias are expressed as a percentage of the quality requirement. In effect, the equation above is normalized by dividing through by TEa, as follows:

Biasobs/TEa = 1 - (?SEcrit + 1.65)*SDobs/TEa

In practice, the normalized OPSpecs charts are scaled from 0 to 100% on the y-axis and 0 to 50% on the x-axis and the x-coordinate and y-coordinate are expressed as a percentage of the quality requirement, as shown in the figure below. This is equivalent to multiplying by 100, as shown below:

100 * Biasobs/TEa = 100 - (?SEcrit + 1.65)*SDobs/TEa*100

where the y-coordinate of the operating point is 100 * Biasobs/TEa and the x-coordinate is SDobs/TEa*100.

Procedure [17]

1. Obtain normalized OPSpecs charts from www.westgard.com, which include common control rules for N=2, 3, 4, and 6, with both 90% AQA and 50% AQA;
2. Define the quality requirement in terms of TEa;
3. Obtain estimates of precision and bias of measurement procedure;
4. Calculate the normalized operating point;
a. Y-coordinate = (Biasobs/TEa) * 100
b. X-coordinate = (SDobs/TEa)*100.
5. Plot the operating point on the normalized OPSpecs chart.
a. Start with low N (2 or 3) with 90%AQA(SE).
b. If no selection, go to high N (4 or 6) with 90%AQA(SE)
c. If no selection, go to high N (4 or 6) with 50% AQA(SE)
d. If no selection, use maximum QC
6. Identify Total QC strategy on basis of available error detection

Advantages

  1. Readily available via the Internet;
  2. More choices than with Sigma-metrics QC planning tool;
  3. More quantitative than Sigma-metrics QC planning tool;
  4. Useful for setting process specifications, as well as selecting QC procedures;
  5. Also provides guidance for Total QC strategy;

Disadvantages

  1. Limited to use of analytical quality requirement of form TEa;
  2. More difficult to understand than Sigma-metric QC design tool;
  3. Documentation is manual process, though facilitated by use of worksheets;

3. EZ Rules® 3 Computer Program

The QC Validator® program, which was the precursor to the EZ Rules® program, has been described in the literature [18] and the automatic selection technology has been validated by comparison with manual applications [19]. The latest version of the program, EZ Rules® 3, is described and illustrated in detailed applications in the 2nd edition of Six Sigma Quality Design and Control [20]. This version supports the use of analytical, biologic, and clinical types of quality requirements, as well as multiple QC designs for “startup,” “monitoring,” and “patient data” QC. The program operates in two modes – an “interview mode” for learning the program through step-by-step entry of parameters and a “form mode” for efficient entry of parameters on a single input screen, as illustrated in the figure below:

ess94f4

Procedure

  1. Name the file for the application;
  2. Enter the parameters on the form;
  3. Review the selected QC procedure;
  4. Document the selection with the QC Validation report.

Advantages

  1. Interview mode of operation facilitates learning program operation, as well as understanding critical input parameters;
  2. Form mode of operation provide efficiency for highly skilled analysts;
  3. Supports biologic goals and clinical decision interval types of quality requirements, as well as analytical allowable total error;
  4. Provides many candidate QC procedures, including mean/range procedures, single-rule variable-limit procedures, as well as single-rule and multi-rule procedures;
  5. Supports “average of normals” patient data QC selection and design;
  6. Supports multistage designs to maximize costs-effective operation of analytical systems;
  7. Provides complete documentation including OPSpecs charts and Sigma-metrics graphs, hardcopy and/or electronic; graphics exportable to other programs for user’s own customized documentation.

Disadvantages

1. Cost (this software is not free);

2. Does not support cumulative sum and exponentially weighted moving average (EWMA) types of QC procedures;

4. Bio-Rad Westgard Advisor Program

This program is scheduled to be introduced to the US marketplace during 2006. Bio-Rad has acquired the intellectual property rights to the automatic QC selection process developed by Westgard QC. This automatic selection process is integrated with the estimates of analytical performance available from Bio-Rad’s well-established peer-comparison database, thereby facilitating the application and minimizing the work required by the laboratory. In effect, the peer-comparison service can now include recommendations for optimizing the QC procedures for individual tests in an individual laboratory based on the performance data from that laboratory.

This program makes use of the OPSpecs chart, as illustrated below:

ess94f5

Obviously, the big advantage with this program is that a laboratory will be able to perform QC planning with minimal effort. In effect, all the tests in the laboratory can be evaluated and assessed based on the data available from participation in the peer-comparison program. Method imprecision will be determined from the monthly QC data that is submitted; method bias will be determined by comparison of the laboratory’s mean values with those of selected peer-groups.

We plan to provide more information on this program once it becomes available.

References

1. Westgard JO, Groth T, Aronsson T, Falk H, deVerdier C H. Performance characteristics of rules for internal quality control: Probabilities for false rejection and error detection. Clin Chem 1977;23:1857 67.
2. Westgard JO, Barry PL, Hunt MR, Groth T. A multi rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981;27:493 501.
3. Westgard JO, Groth T. Design and evaluation of statistical control procedures: Applications of a computer 'QC Simulator' program. Clin Chem 1981;27:1536 1545.
4. Westgard JO, Groth T. Power functions for statistical control rules. Clin Chem 1979;25:863 69.
5. Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the quality and productivity of analytical processes. AACC Press, Washington, DC, 240 p, 1986.
6. Koch DD, Oryall JJ, Quam EF, Feldbruegge DF, Dowd DE, Barry PL, Westgard JO. Selection of medically useful quality-control procedures for individual tests done in a multitest analytical system. Clin Chem 1990;36:230-3.
7. Westgard JO, Hytoft Petersen P, Wiebe DA. Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program (NCEP). Clin Chem 1991:37:656-661.
8. Westgard JO, Wiebe DA. Cholesterol operational process specifications for assuring the quality required by CLIA proficiency testing. Clin Chem 1991;37:1938-44.

Sigma-metrics QC Planning Tool

9. Westgard JO, Groth T, deVerdier C-H. Principles for developing improved quality control procedures. Scand J clin Lab Invest 1984;44(suppl 172):19-41.
10. Westgard JO. Six Sigma Quality Design and Control: Desirable precision and requisite QC for laboratory measurement processes, 2nd ed. Madison WI:Westgard QC, 2006.
11. Westgard JO. Quality Control: How labs can apply Six Sigma principles to quality control planning. Clin Lab News 2006(Jan):10-12.
12. Westgard JO, Ehrmeyer SS, Darcy TP. “Doing the Right QC”, Chapter 13 in CLIA FINAL RULES: Quality Assessment Issues and Answers. Madison WI:Westgard QC, Inc., 2004.
13. CLSI C24-A3. Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definition; Approved Guideline – Third Edition. 2006. Clinical and Laboratory Standards Institute, 940 West Valley Road, Suite 1400, Wayne, PA 19087-1898 USA.

OPSpecs Charts

14. Westgard JO. Charts of Operational Process Specifications (OPSpecs Charts) for assessing the predicione, accuracy, and quality control needed to satisfy proficiency testing criteria. Clin Chem 1992;38:1226-33.
15. Westgard JO. Assuring analytical quality through process planning and quality control. Arch Pathol Lab Med 1992;116:765-9.
16. Westgard JO. OPSpecs Manual – Expanded Edition. Madison WI:Westgard QC, 1996.
17. Westgard JO. Basic Planning for Quality. Madison WI:Westgard QC, 2000.

EZ Rules Computer Program

18. Westgard JO, Stein B, Westgard SA, Kennedy R. QC Validator 2.0: a computer program for automatic selection of statistical QC procedures in healthcare laboratories. Comput Method Program Biomed 1997;53:175-186.
19. Westgard JO, Stein B. An automatic process for selecting statistical QC procedures to assure clinical or analytical quality requirements. Clin Chem 1997;43:400-403.
20. Westgard JO. Six Sigma Quality Design and Control: Desirable precision and requisite QC for laboratory measurement procedures. 2nd ed. Madison WI:Westgard QC, 2006.

James O. Westgard, PhD, is a professor of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.