SIX SIGMA - QUALITY DESIGN AND CONTROL APPLICATIONS
James O. Westgard, PhD
SIX SIGMA -
This article illustrates several applications of the OPSpecs chart as a Six Sigma quality design and control tool. It assumes that you have some familiarity with the OPSpecs chart - if you don't, feel free to check in the archives for a multitude of lessons. (Adopting the OPSpecs Chart as Your Planning Tool, in particular, is very helpful)Six Sigma Quality Management can be combined with the use of OPSpecs charts to do the following:
- set performance specifications,
- select QC procedures,
- identify method improvement strategies,
- assess the benefits of method improvements,
- compare clinical and analytical quality requirements, and
- relate proficiency testing criteria to clinical quality requirements.
To get started, you need to know how to read an OPSpecs chart. If you are already familiary with them, you can skip this material.
Here’s a figure that points out all the critical information on the OPSpecs chart – the ABC’s (and D) for reading the chart. Start with the title of the chart, then look at the axes of the chart, then the grid area of the chart, and finally the key on the right side of the chart. If all of the terms look familiar and are understandable, you can skip to the next section. For more details about the chart, read on.
A. Start at the top with the title line:
- OPSpecs refers to operating specifications. There also are power function graphs and critical-error graphs that may be encountered along with OPSpecs charts.
- Dint indicates the type of quality requirement for which the chart was prepared. Dint refers to a clinical quality requirement in the form of a decision interval or medically important change in a test result. You will also see charts for TEa, which refers to an analytical quality requirement in the form of an allowable total error.
- 20% is the actual value for the decision interval. The units are given as percent for several key parameters in the chart – the quality requirement, allowable inaccuracy on the y-axis, and allowable imprecision on the x-axis.
- 90% is the level of Analytical Quality Assurance (AQA), or the error detection capability given again in %. The value of 90% corresponds to a probability of error detection of 0.90. Charts may also be prepared for 50% and 25% AQA (SE). SE indicates this chart was prepared for the detection of systematic error. It is also possible to prepare OPSpecs charts for random error (RE).
B. Look at the axes of the chart.
- The y-axis shows the inaccuracy or bias of the method or measurement procedure, biasmeas, in units of %, i.e., relative to a concentration that is critical for interpretation of the test result, which is also called a medical decision level, Xc.
- The x-axis shows the imprecision or standard deviation of the method or measurement procedure, smeas, in units of %, i.e., a coefficient of variation or CV.
C. Look in the grid area of the chart
- The operating point shows the imprecision (x-coordinate) and inaccuracy (y-coordinate) of the method under study. In this example, the method has a CV of 3.0% and a bias of 0.0%, giving an operating point that lies 3 units out on the x-axis.
- Each line describes the limits of bias and CV that are allowable for a given QC procedure. The lines show the operating limits for the QC procedure and may also be referred to as operating lines.
- The QC procedures for each line, appearing top to bottom in the chart, are identified in the key at the right, also listed in order top to bottom.
D. Look at the key.
- The control rules are shown in the 1st column using abbreviations of the form 13s, which indicates a run will be judged out-of-control if 1 control measurement exceeds control limits set as the mean ± 3s. Similarly, 22s is a control rule for 2 consecutive control measurements exceeding the same mean + 2s limit or the same mean – 2s limit. R4s refers to a range rule, where the difference between the high and low measurements in a group of controls exceeds 4s; this rule may also be implemented as a counting rule, where one control measurement exceeds a + 2s limit and another exceeds a –2s limit. 41s refers to 4 consecutive measurements exceeding the same mean + 1s or mean – 1s limit.
- Probability for false rejection is shown in the 2nd column. This is actually given as a probability, not a percent.
- N refers to the total number of control measurements considered in making the assessment of control status. N=4 could mean that two materials are each analyzed twice or four different materials are each analyzed once.
- R refers to the number of runs over which the data are interpreted. Usually R will be 1, which indicates all the rules are applied in a single run. If R is greater than 1, it indicates that at least one rule is being applied to data across two or more runs, e.g., if an 8x rule (eight on one side of the mean) were being used with N=4, it would be necessary to have R=2 to accumulate 8 control measurements.
To demonstrate the many applications of OPSpecs charts, I’ll use my favorite test – cholesterol. It’s my favorite because there are national guidelines for interpretation of the test and national standards for the performance of the method. The U.S. CLIA regulations define 10% as the acceptability criterion or allowable total error in a proficiency testing event [1]. The National Cholesterol Education Program (NCEP) provides performance specifications of 3.0% maximum bias and 3.0% maximum CV for a cholesterol method [2]. For comparison, the European biologic goals are 4.1% for the maximum bias and 2.7% for the maximum CV [3]. NCEP also provides national interpretative guidelines that recommend values of 200 mg/dL or lower are okay while values of 240 mg/dL and higher require additional followup, usually additional tests, to formulate a treatment plan. This is a good illustration of a clinical decision interval or medically important change in a test result. A patient whose true homeostatic set-point is 200 mg/dL should never end up with a measured value of 240 mg/dL because that will lead to mistreatment. A change of 40 mg/dL at a level of 200 mg/dL, or 20%, defines a clinical decision interval. It’s important to understand that this interval includes preanalytical variables, such as the subject’s own biological variation, which corresponds to a 6.5% standard deviation for the average subject [4].
To demonstrate the applications of OPSpecs charts for the cholesterol tests, appropriate charts were prepared as described in Chapter 10. Those charts are presented here without any further discussion of their preparation. The focus here is on the questions that can be answered with the information provided by OPSpecs charts. The format is to state the question, show the appropriate OPSpecs chart, then discuss the answer.
It’s a fair assumption that most laboratories in the U.S. run only two controls for cholesterol because that’s all that’s required by CLIA. If we look at the commonly used control rules and Ns of 2, as shown in the chart above, the x-intercepts of the lines define the maximum allowable CV when bias is zero. For example, if the common practice is to use a Levey-Jennings control chart with control limits set as the mean plus and minus 3 standard deviations, then the 2nd line from the bottom shows that the maximum allowable CV is about 2.4%. If the method bias were specified to be 2.0%, then the allowable CV would be 2.0%, as shown at the point of intersection with the operating line.
The generic answer to this type of question requires preparation of an OPSpecs chart with the quality requirement of interest and the QC procedures expected to be found in routine practice. The answer is given by the x-coordinate at a specified bias on the line representing the control rules and number of control measurements that are expected in practice. When the specified bias is zero, the maximum allowable CV is given by the x-intercept of the line.
If a laboratory implements a method having a CV of 3.0% and a bias of 3.0%, then the typical laboratory QC practice of using 2 measurements per run will not be sufficient. The chart above shows an operating point having a y-coordinate of 3.0% and an x-coordinate of 3.0% that represents the maximum allowable specifications according to NCEP. Only one line is above the operating point, therefore there is only one choice for QC – a multirule procedure having a total of 6 control measurements per run.
The general answer to this type of question is found by preparing an OPSpecs chart for the quality requirement of interest, then plotting the method CV and bias to locate the operating point on the chart. You then try different candidate QC procedures, starting with low N, then increasing N and formulating multirule procedures, until sufficient error detection is obtained, i.e., the operating line of a QC procedure is above the operating point of the method.
You can assess the effect of improvements in method performance by relocating the operating point. For example, if the method bias could be reduced to zero, the operating point would be located as shown on the chart above. It would then be possible to assure the quality of the testing process by using 12.5s control rule with 3 control measurements per run or a 13s/2of32s/R4s/31s multirule procedure with 3 control measurements per run.
The generic solution is to prepare the OPSpecs chart for the quality requirement of interest and the operating point that represents current method performance. Then move the operating point down to the x-axis so it now represents the method CV and zero bias. See what QC procedures would become effective for the relocated operating point.
You can assess the additional effect of improving the method CV by moving the operating point to the left along the x-axis, as shown on the OPSpecs chart above. If bias were zero, the method’s CV would need to be reduced to 2.4% or less in order to reduce the number of control measurements to 2 per run, which would provide additional cost savings in the routine operation of this cholesterol testing process.
The general solution is to prepare an OPSpecs chart that shows the QC procedures of interest along with an operating point showing current method performance. Then identify where the operating point needs to be located, this time moving it horizontally to the left, until it falls within the allowable limits for a QC procedure. That’s the improvement needed in the method’s CV to be able to change to a different QC procedure.
For cholesterol, the U.S. has national guidelines for both clinical and analytical quality requirements. The NCEP guidelines for test interpretation correspond to a clinical decision interval of 20%. The CLIA criterion for acceptable performance in proficiency testing events is an allowable total error of 10%. You might reasonably assume that these quality requirements are coherent or consistent, i.e., that analytical requirement for proficiency testing represents the clinical performance or medical usefulness of the test.
The two OPSpecs charts on the opposite page show the comparison of the allowable imprecision for the clinical requirement (top chart) with that for the analytical requirement (bottom chart). For the same set of candidate QC procedures, the allowable CV is 2.4% to 4.3% for the clinical requirement, but only 1.9% to 3.0% for the analytical requirement. Clearly, the proficiency testing criterion sets a more demanding requirement for method performance than is needed for medical usefulness , which strongly suggests that current U.S. requirements for quality are incoherent!The general answer to this type of question is found by preparing two OPSpecs charts, one for each of the quality requirements of interest, but both with the same candidate QC procedures. The method CVs that are allowable can be read from the x-intercepts of the two charts, then compared to see which is more demanding.
It makes no sense that the CLIA criterion for acceptable performance in proficiency testing events should be more demanding that the performance needed for medical usefulness. After all, the purpose of performing laboratory tests is not to pass proficiency testing, but to provide medically useful test results for patient treatment. The problem is that clinical and analytical quality requirements are of different formats and can’t be directly compared numerically. They’re like apples and oranges, both fruit, but different colors and different tastes. Which is better or more correct? How can you compare them?
The OPSpecs charts above provide an answer. The top chart is for a clinical decision interval of 20%. The bottom chart was prepared for an allowable total error of 13%. Both show the same candidate QC procedures, but note that the scaling of the axes are not the same. The x-intercepts for those QC procedures show a range of allowable method CVs from 2.4% to 4.2% for the clinical requirement and a range of 2.5% to 3.9% for the analytical requirement. A CLIA criterion of 13% would more properly represent the medical usefulness of a cholesterol test.
The generic solution is to first prepare the clinical OPSpecs chart for a selected group of candidate QC procedures. Then prepare an analytical OPSpecs chart and vary the TEa input parameter until the range of allowable CVs approximately matches the range observed for the clinical chart.
Current specifications for method performance are inadequate! A laboratory method should have a CV of 2.0% or better if bias is near zero; an even smaller CV is needed when bias is not zero. Compare these specifications to the NCEP maximum allowable CV of 3.0% and maximum allowable bias of 3.0%. There must be a significant difference in process capability based on these specifications.
The inadequacy of the NCEP performance specifications can be understood by calculating the actual sigma-metric from the NCEP tolerance limits (interpretative guidelines and the resulting clinical decision interval). As we've discussed earlier in other Six Sigma lessons, the critical systematic error that is calculated from the tolerance limits and performance specifications can be converted to a sigma-metric by adding the z-value that was used in the calculation of that error. In this case, the critical SE is 1.73, as shown in the top graph on the opposite page. The process sigma is 1.73 plus the z-value of 1.65, or 3.38-sigma, which is near the minimum that is acceptable for routine operation of any production process. The European biologic goals of 2.7% maximum CV and 4.1% maximum bias provide a very similar process metric of 3.24-sigma (1.59 + 1.65).
The difficulty of controlling a process with 3-sigma capability can be seen from the critical-error graph. The intersection of the critical-error line (the vertical line) with the power curves for commonly used QC procedures occurs on the rising portion of the power curves, resulting in probabilities of error detection from 0.20 to 0.60, or 20% to 60%, for Ns up to 4. It takes a multirule procedure with N=6 to provide 90% detection of the critical SE.
The process capability for the CLIA tolerance limits is even more dismal, as shown by the figure at the bottom of the page. The critical SE is only 0.68, which corresponds to a process capability of only 2.33-sigma (0.68 + 1.65). Such a process is inadequate for a routine service operation.
On the other hand, a method whose CV is 2.0% and bias is 0.0% provides 6.04-sigma performance for the NCEP clinical tolerance limits and 5.00-sigma performance for the CLIA analytical tolerance limits. Those specifications are consistent with the performance goals in Six Sigma Quality Management.
- U.S. Department of Health and Human Services. Medicare, Medicaid and CLIA Programs; Regulations implementing the Clinical Laboratory Improvemenet Amendments of 1988 (CLIA). Final rule. Fed Regist 1992;57:7002-186.
- National Cholesterol Education Program Laboratory Standardization Panel. Current status of blood cholesterol measurements in clinical laboratories in the United States. Clin Chem 1988;34:193-201.
- Fraser CG, Hyltoft Petersen P, Ricos C, Haekel R. Proposed quality specifications for the imprecision and inaccuracy of analytical systems for clinical chemistry. Eur J Clin Chem Clin Biochem 1992;30:311-317.
- Costongs GMPJ, Janson PCW, Bas BM, et al. Short-term and long-term intra-individual variations and critical differences of clinical chemical laboratory parameters. J Clin Chem Clin Biochem 1985;23:7-16.
