Tools, Technologies and Training for Healthcare Laboratories
  2. Home
  3. Essays
  4. QC Design
  5. Quality by Design

QC Design

Quality by Design

If we're honest with ourselves, we know that the situation in the laboratory needs to be improved. But with all the time pressures, staff shortages, and other stresses, how can we find a quick way to put the quality back into quality control? How do we select methods that can guarantee error detection? Create purchase specifications to select instruments that can guarantee we meet our CLIA requirements? Design instruments with quality in mind? Dr. Westgard suggests practical steps that both users and manufacturers can take to bring quality into their design.

An updated version of this essay appears on the Nothing but the Truth about Quality book.

1997

An updated version of this essay appears on the Nothing but the Truth about Quality book.

One of the themes of the recently published Clinical Chemistry Forum on "Quality for Tomorrow" is that quality should be designed or built-in to a process or product [1]. That certainly is fundamental to our work on quality planning. It is also the reason for determining operating specifications that describe the imprecision and inaccuracy that are allowable and the QC that is necessary to assure a desired clinical quality requirement will be achieved in routine laboratory testing.

Another theme of this Forum seems to be that statistical QC is no longer relevant because most of the errors or problems now occur in pre- or post-analytical steps. While the data to support this premise are minimal, many people want to believe that analytical performance is no longer a problem. Such data would certainly suggest that additional efforts are needed to design quality into these pre- and post-analytical steps, but we should not infer that statistical QC is no longer necessary or useful. It is very likely that statistical QC is one of the reasons for the low rate of analytical errors, thus quality systems need to build on the success and capability of statistical QC.

This second theme probably reflects the hopes and wishes of manufacturers and users of Point-of-Care (POC) testing devices. In POC settings, statistical QC is often difficult to implement and costly to perform, yet these are the settings that would be expected to be susceptible to pre- and post-analytical problems, as well as analytical problems. Quality by design is certainly important for minimizing these potential problems, but statistical QC also is important for providing an independent check of a wide variety of variables and for monitoring the skills of operators. One might wonder if "Quality by Design" wouldn't avoid the POC setting if the financial incentives weren't so great.

A manufacturer's approach - "design control"

In one manufacturer's discussion of a rigorous approach to "design control" [2], it was stated that "...the goal is to deliver products acceptable for their intended use." This discussion also conceded that "clinical performance standards for IVD tests have not been generally available [and] few examples are published in the peer reviewed literature." One wonders then what level of quality is being designed into the products and how users can be sure these products meet the clinical needs of their patients!

The use of "clinical performance standards" in this discussion [2] provides an example of the difficulties in the terminology and formats for quality standards, and extends the list of terms mentioned in my earlier discussion of "Quality Goals, Requirements, and Specifications". "Clinical performance standards" is another ill-defined term, and although it has a nice ring to it, it's not at all clear that clinical has anything to do with what is really used. Rather, it was stated that manufacturers often use interlaboratory performance data from proficiency testing surveys instead of clinical performance standards because these interlaboratory analytical performance data are more readily available. Thus, standards of quality are still an issue, even when manufacturer's have a rigorous approach for design and production.

A user's approach - "specifications for purchase"

Since laboratories and POC users mainly buy laboratory methods and systems that are designed by manufacturers, it seems that users must trust manufacturers to know what quality is needed and to deliver products that assure that level of quality. Alternatively, users can establish "purchase specifications" for imprecision and inaccuracy that will assure the methods and systems obtained from manufacturers have the necessary analytical performance. "Analytical performance" is the right information here because product labelling generally requires claims for the imprecision and inaccuracy of the method or system, therefore, this information is generally documented and available to users.

Define a quality requirement To determine purchase specifications, users can first define the quality that is required for their applications and then translate that quality requirement into operating specifications for the imprecision and inaccuracy of the method and the QC needed to monitor method performance. Quality planning models are used to define the relationships between these parameters, then OPSpecs charts can be used to provide a graphical display of the allowable inaccuracy (on the y-axis) and allowable imprecision (on the x-axis) for a QC procedure (specific control rules and number of control measurements) that is practical to use in the laboratory.

Prepare an OPSpecs chart OPSpecs charts can be prepared for analytical quality requirement in the form of an allowable total error, clinical quality requirements in the form of a medically important change or decision interval, or even with biological goals for imprecision and inaccuracy (which are discussed by Per Hyltoft Petersen in his essay on European approaches to analytical goal setting). If purchase specifications are established from an OPSpecs chart, users can obtain methods whose performance can be assured to meet the desired quality in routine operation by monitoring with statistical QC procedures.

Determine the x-value from the OPSpecs line When the allowable inaccuracy or bias is zero, the maximum allowable imprecision is determined by the x-intercept of the OPSpecs line for the QC procedure(s) of interest. For a given level of inaccuracy or bias, the corresponding allowable imprecision can be read along the OPSpecs line as the x-value that corresponds to the y-value for the allowable bias.

State the purchase specifications In general, it is best to state the maximum allowable imprecision at a bias of zero. Then add a note to the effect that if bias is not zero, imprecision will need to be better. How much better depends on the size of the bias and can be determined from the OPSpecs line by identifying the x-coordinate corresponding to observed bias (y-coordinate).

Example application for cholesterol

For an analytical quality requirement of 10%, the accompanying OPSpecs chart shows the operating limits for commonly used control rules with Ns of 2. The x-intercepts show that the maximum allowable imprecision (for bias of 0.0) is from 1.7 to 2.4%. Note that the 2.4% figure is for the 12s rule, which has a probability of false rejection of 0.09, meaning that 9% of the runs would be rejected even if everything were working perfectly. This high level of false rejections would be too costly in most laboratories, so it would be better to set the maximum allowable imprecision at about 2.1% (for the 12.5s rule with N=2 which has about 3% false rejections). Purchase specifications should require a CV of 2.1% or better (with a bias of 0.0%) to assure that the method will be controllable in routine laboratory operation by use of a simple single-rule, low-N QC procedure such as 12.5s with N=2.

For the clinical quality requirement established by the National Cholesterol Education Program (NCEP), the clinical OPSpecs chart for a decision interval of 20% (after accounting for a within-subject biological variation of 6.5%) shows the maximum allowable imprecision by the x-intercepts from 2.1 to 3.2%. A practical maximum is 2.7% for the 12.5s rule with N=2 (again discarding the 12s rule because of its high false rejection rate of 9%). Purchase specifications should require a CV of 2.7% or better (with a bias of 0.0) to assure that the method will be controllable to the quality required for clinical use and interpretation of cholesterol as a screening test.

Comparison with NCEP specifications

NCEP has recommended that the maximum allowable imprecision should be 3% and the maximum allowable inaccuracy should be 3% [3]. A method that satisfies the NCEP specifications clearly doesn't satisfy the purchase specifications recommended here (2-2.5% imprecision, 0.0% inaccuracy). Note that an operating point that represents the NCEP specifications would satisfy the bias + 2SD criterion for stable performance, as shown on the accompanying OPSpecs chart. This suggests that the underlying premise of the NCEP specifications is the total error model that incorporates the stable performance of the method, but does not consider the sensitivity of a QC procedure for detecting unstable errors; this simple model provides a design objective of about "three sigma" (10/3). Manufacturers would argue that the industrial design objective is "six sigmas"[2], which means that a cholesterol method with an error tolerance of 10% should have a CV of 1.7% (10/6). Thus, purchase specifications of 2.0 to 2.5% should be achieveable given manufacturers' design objectives.

User's vs manufacturer's strategies for quality by design

The industrial strategy is to establish a design objective such as "six sigma" process capability, then to improve the process (reduce its variability or standard deviation) until the desired performance is achieved. Given processes with a uniform high level of process capability, standard statistical QC procedures, such as 3 SD control limits, can generally be used for process control. Thus, manufacturers generally control the process capability and, when "six sigma" capability is achieved, do not have to be concerned about the careful selection of QC procedures.

The users of a testing process generally purchase a method or system and have little ability to change its inherent variability (or process capability). The user's strategy is to evaluate the capability of the process, then provide the necessary control to achieve the desired quality. When multitest analyzers are purchased, it is not realistic to expect that all tests will have the same level of process capability, thus the user must adjust the QC to fit the process capability observed of the method and the quality required for patient care, rather than adjusting the process capability to use a standard QC practice. In setting purchase specifications and evaluating method performance, this means that users must set more demanding criteria, such as a 4 SD total error criterion (bias + 4 SD < TEa) or preferably a 5 SD criterion (bias + 5 SD < TEa). [See reference 4 for a discussion of process capability and its relationship to method acceptance criteria.]

The OPSpecs chart as a user's tool for quality by design

OPSpecs charts with 90% AQA require approximately four to five sigma process capability if low N QC procedures are to provide the necessary error detection, thus OPSpecs charts provide a user's "design control" tool that is not quite as demanding as the industrial "six sigma" design objective. This equivalence has been demonstrated by comparing the operating limits of commonly used QC procedures with lines corresponding to different total error criteria [5]. OPSpecs charts, therefore, can be used directly for judging the acceptability of method performance for analytical, clinical, or biologic quality requirements, and can likewise be used to establish analytical performance specifications for purchase of methods and systems.

References

  1. Garber CC, Witte DL. Quality for tomorrow: by design or by checking? Clin Chem 1997;43:864-865.
  2. Lasky FD, Boser RB. Designing in quality through design control: a manufacturer's perspective. Clin Chem 1997;43:866-872.
  3. National Cholesterol Education Program Laboratory Standardization Panel. Current status of blood cholesterol measurements in clinical laboratories in the United States. Clin Chem 1988;34:193-201.
  4. Westgard JO, Burnett RW. Precision requirements for cost-effective operation of analytical processes. Clin Chem 1990;36:1629-1632.
  5. Westgard JO. A method evaluation decision chart (MEDx chart) for judging method performance. Clin Lab Science 1995;8:277-283.