SIX SIGMA -
PATIENT DATA FOR ASSESSING PROCESS PERFORMANCE AND STABILITY

James O. Westgard, PhD

• What's the Patient Data/Average of Normals(AoN)?
• Potential usefulness of AoN applications
• Performance vs Process Sigma Metric
• Performance vs s_pop/s_meas ratio
• Necessary or available number of patient samples
• Applicability in a Specific Laboratory
• Design of AoN algorithms
• What's the point?
• References

This application illustrates an advanced combination of Quality Control techniques: using both Sigma-metrics and Patient Data QC (also known as Average of Normals, AoN, Average of Patients). Patient Data QC uses patient test results to assess the performance of a laboratory testing process and to identify changes in process stability. This application strategy fits together with the strategy of multistage quality control, which employs multiple QC designs to accomplish specific objectives. In the ideal world, a QC procedure should provide high detection of medically important errors, low false rejection, and low cost, i.e, require only a few measurements. When ideal performance cannot be achieved with a single QC procedure, then multiple QC designs can be employed to accomplish specific purposes. The generic structure for a multi-stage laboratory QC system would have three stages:

• Startup QC for high error detection;
• Monitor QC for low false rejection;
• Patient data QC for measurement of run length.

This strategy generally becomes applicable when the process capability is less than 5-sigma and usually is necessary when less than 4-sigma. Even processes having less than 3-sigma performance may be adequately monitored if the patient data QC can be properly designed.

What's the Patient Data/Average of Normals?

The utilization of patient data for process control is of much interest in the new millennium. The 2000 annual meeting of the American Association for Clinical Chemistry featured an Edutrak session titled "The use of patient data for process control: It's time has come" [1]. Actually, patient data algorithms have been of interest for a long time, beginning in 1965 with Hoffman and Waid's "average of normals" (AoN) technique [2]. This technique makes use of patient test results that fall within "normal" range, or within truncation limits that approximate the reference range for the test, to calculate an average that is used to monitor the stability of the process. The assumption is that large groups of patient test results should demonstrate essentially the same mean value, within the limits expected from the standard error of the mean. A change in the mean value suggests that a change has occurred in the process and that its operation no longer represents stable performance.

In 1984, Cembrowski et al [3] utilized computer simulation to determine power curves that described the error detection characteristics of AoN procedures and to demonstrate that AoN performance depended on the ratio of patient population standard deviation to the method standard deviation, the width of the truncation limits, the number of patient test results remaining after truncation, and the width of the control limits for the mean patient value. More recently, a quantitative design approach has been developed with the support of the OPSpecs design tool [4].

The purpose of this chapter is to illustrate the potential usefulness of AoN algorithms for monitoring process performance, assessing changes or instability in the process, and signaling when the process needs to be re-validated. For laboratory tests where process capability is low, an AoN algorithm may provide greater error detection than available from QC using stable control materials. It may also provide a measure of run length to identify when new control samples should be tested to validate the performance of the process. For such an application, AoN algorithms could be used even with 6-sigma processes to minimize the cost of operation.

Potential usefulness of AoN applications

The usefulness of AoN algorithms depends on quality requirement for the test, the analytical performance of the method, the ratio of the population standard deviation (s_pop) to analytical standard deviation (s_meas), and the number of patient test results (subgroup n) that are available in the particular healthcare laboratory. The design process is similar to that for traditional statistical QC procedures, but must consider the additional s_pop/s_meas variable. The effectiveness of an application can be judged from the expected probability of error detection (P_ed).

To demonstrate the potential usefulness of AoN applications, I'll make use of the study data and criteria for acceptable performance from the Laboratory Proficiency Testing Program (LPTP) of the Ontario Medical Association, method performance characteristics representative of state of the art automated systems, and a patient populations of a regional reference laboratory[5]. The sigma metrics for 37 tests were calculated earlier and take into account the LPTP criteria and the observed method performance. Additional information on the s_pop/s_meas, subgroup n needed or available, and the expected P_ed are shown in the following tables.

Performance vs Process Sigma Metric

The table below shows the tests in the order of process capability, top to bottom from the highest sigma-metric of 10.0 to the lowest value of 1.67. The probabilities for error detection, as shown in the column at the right side of the table, generally correlate with the sigma metric, but not always.

Table 1. Observed sigma metric and the applicability of Average of Normals (AoN) patient data algorithms for each test and the available number of patient specimens (subgroup n) in a regional reference laboratory.

Test Name	Sigma metric	spop/smeas	Subgroup n	Ped
Bilirubin, total	10.0	8.60	120	>0.96
Hematocrit	10.0	5.00	80	1.00
Hemoglobin	10.0	9.09	120	>0.96
MCV	10.0	9.18	120	>0.96
RBC	10.0	9.71	120	>0.96
Lipase	10.0	12.7	60	>0.14
Platelets	7.83	11.6	180	>0.96
Amylase	7.50	8.54	60	>0.63
Urate	6.67	13.6	300	>0.97
ALT	6.18	5.37	40	>0.98
AST	6.00	2.20	40	1.00
Creatinine	5.56	4.44	40	0.99
Phosphate	5.56	7.65	90	0.93
ALP	5.42	13.8	450	0.99
Glucose	5.38	6.00	90	0.99
LD	5.33	6.59	60	0.80
T3, total	4.80	2.18	10	0.94
T3 uptake	4.80	3.00	20	0.76
T4, total	4.80	6.10	60	0.18
TSH	4.62	8.88	300	0.99
CK	4.50	4.93	20	0.31
LH	4.36	15.7	180	0.27
Bicarbonate	4.35	3.48	40	0.96
Urea	4.26	11.1	450	0.91
Ferritin	4.00	7.89	120	0.94
Chloride	3.57	2.58	30	1.00
T4, free	3.43	2.52	100	0.99
FSH	3.43	19.7	180	0.27
Prolactin	3.43	6.33	120	0.56
Potassium	3.33	2.78	100	0.98
Vitamin B12	3.00	13.8	120	0.03
Calcium	2.86	1.84	60	0.91
Protein, total	2.63	4.94	80	0.09
Triglyceride	2.63	32.5	600	0.19
Sodium	2.50	1.58	60	0.93
Cholesterol, total	1.91	10.3	450	0.12
Cholesterol, HDL	1.67	7.00	300	0.16

Key Observations:

• For tests whose sigma metrics are 6 or greater, P_ed is generally 0.90 or higher, which is considered ideal performance. Note, however, that the lipase test having 10-sigma performance does not lend itself to effective monitoring with patient data. The AoN algorithm provides a P_ed of only 0.14, or 14% detection of medically important errors. This is explained by the low number of patient samples that are available for daily testing (only 60) and the high ratio of population to analytical variation (12.7). A similar situation exists for amylase, but the lower s_pop/s_meas ratio (8.54) makes for better error detection - a 60% chance of detecting medically important errors.
• Tests having sigma metrics of 4 and 5 generally show good error detection with AoN algorithms, but again there are exceptions that can be related to the s_pop/s_meas ratio or the limited number of patient samples available on a daily basis. There also are some surprises such as total T3 which requires only 10 patient samples to achieve 94% detection of medically important errors due to the low s_pop/s_meas ratio of 2.18 and its high 4.80-sigma metric for process performance.
• Tests having sigma-metrics of 3 and lower show very mixed performance with P_ed values from 0.09 to 1.00. Note that sodium, whose process performance is only 2.50-sigma, can be effectively monitored by an AoN algorithm utilizing 60 patient test results. Similarly, calcium with 2.86-sigma performance can also be effectively monitored with an AoN algorithm and 60 patient test results. The reasons, of course, are the low s_pop/s_meas ratios of 1.58 for sodium and 1.84 for calcium.

There certainly is a correlation between the sigma-metric of the process and the expected AoN error detection, but each test must be examined carefully. It is possible that tests with very low process capability, but very low s_pop/s_meas ratios, can be more effectively monitored by patient data than by traditional stable control materials. The multi-stage QC strategy is clearly advantageous for these applications.

Performance vs s_pop/s_meas Ratio

The table below shows the tests ordered top to bottom by their s_pop/s_meas ratios. It is expected that the lower the ratio, the less population variation, the tighter the distribution of patient values, and the lower number of patient samples that would be needed to provide an estimate of the mean value.

Table 2. Observed ratio of population to measurement variation (s_pop/s_meas ratio) and the applicability of Average of Normals (AoN) patient data algorithms for each test and the number of patient specimens (subgroup n) in a regional reference laboratory.

Test Name	Sigma metric	spop/smeas	Subgroup n	Ped
Sodium	2.50	1.58	60	0.93
Calcium	2.86	1.84	60	0.91
T3, total	4.80	2.18	10	0.94
AST	6.00	2.20	40	1.00
T4, free	3.43	2.52	100	0.99
Chloride	3.57	2.58	30	1.00
Potassium	3.33	2.78	100	0.98
T3 uptake	4.80	3.00	20	0.76
Bicarbonate	4.35	3.48	40	0.96
Creatinine	5.56	4.44	40	0.99
CK	4.50	4.93	20	0.31
Protein, total	2.63	4.94	80	0.09
Hematocrit	10.0	5.00	80	1.00
ALT	6.18	5.37	40	>0.98
Glucose	5.38	6.00	90	0.99
T4, total	4.80	6.10	60	0.18
Prolactin	3.43	6.33	120	0.56
LD	5.33	6.59	60	0.80
Cholesterol, HDL	1.67	7.00	300	0.16
Phosphate	5.56	7.65	90	0.93
Ferritin	4.00	7.89	120	0.94
Amylase	7.50	8.54	60	>0.63
Bilirubin, total	10.0	8.60	120	>0.96
TSH	4.62	8.88	300	0.99
Hemoglobin	10.0	9.09	120	>0.96
MCV	10.0	9.18	120	>0.96
RBC	10.0	9.71	120	>0.96
Cholesterol, total	1.91	10.3	450	0.12
Urea	4.26	11.1	450	0.91
Platelets	7.83	11.6	180	>0.96
Lipase	10.0	12.7	60	>0.14
Urate	6.67	13.6	300	>0.97
ALP	5.42	13.8	450	0.99
Vitamin B12	3.00	13.8	120	0.03
LH	4.36	15.7	180	0.27
FSH	3.43	19.7	180	0.27
Triglyceride	2.63	32.5	600	0.19

Key Observations:

• The tests in the top half of the show that relatively low numbers of patient samples can be used when the s_pop/s_meas ratio is 6 or less. AoN algorithms can provide at least 90% detection of medically important errors with only 10 samples for total T3, only 30 for chloride, only 40 for AST, ALT, bicarbonate, and creatinine, only 60 for sodium and calcium, and 90 for glucose.
• The application of AoN algorithms is limited by the available number of patient samples for tests such as T3 uptake and CK, whose sigma-metrics are high and s_pop/s_meas ratios are low.
• Tests whose s_pop/s_meas ratios are in the range from 6 to 10 can often be effectively monitored by AoN algorithms using higher numbers of patient samples, usually greater than 100.
• Occasional tests with very high s_pop/s_meas ratios can still be monitored with AoN algorithms when a large number of patient samples are available, e.g., 300 for urate and 450 for ALP.

Clearly, the s_pop/s_meas ratio is a major determinate of the applicability of AoN algorithms. Analytes that are tightly controlled by the body, such as calcium and sodium, provide optimal applications. Those particular tests are difficult to monitor by traditional QC procedures that use stable control samples. Thus, the implementation of AoN algorithms provides a necessary and complementary QC design and further demonstrates the need for the multi-stage QC strategy.

Necessary or available number of patient samples

The table below shows the tests arranged top to bottom in order of the number of patient samples that are needed or available. *Those test names shown in italics indicate situations where the limited availability of patient samples actually limits the application of AoN algorithms in this particular laboratory.

Table 3. Necessary or available patient samples (subgroup n) and the applicability of Average of Normals (AoN) patient data algorithms for each test and the number of patient specimens (subgroup n) in a regional reference laboratory.

Test Name	Sigma metric	spop/smeas	Subgroup n	Ped
T3, total	4.80	2.18	10	0.94
*T3 uptake	4.80	3.00	20	0.76
*CK	4.50	4.93	20	0.31
Chloride	3.57	2.58	30	1.00
AST	6.00	2.20	40	1.00
Creatinine	5.56	4.44	40	0.99
ALT	6.18	5.37	40	>0.98
Bicarbonate	4.35	3.48	40	0.96
Sodium	2.50	1.58	60	0.93
Calcium	2.86	1.84	60	0.91
*LD	5.33	6.59	60	0.80
*Amylase	7.50	8.54	60	>0.63
*T4, total	4.80	6.10	60	0.18
*Lipase	10.0	12.7	60	>0.14
Hematocrit	10.0	5.00	80	1.00
*Protein, total	2.63	4.94	80	0.09
Glucose	5.38	6.00	90	0.99
Phosphate	5.56	7.65	90	0.93
T4, free	3.43	2.52	100	0.99
Potassium	3.33	2.78	100	0.98
Bilirubin, total	10.0	8.60	120	>0.96
Hemoglobin	10.0	9.09	120	>0.96
MCV	10.0	9.18	120	>0.96
RBC	10.0	9.71	120	>0.96
Ferritin	4.00	7.89	120	0.94
*Prolactin	3.43	6.33	120	0.56
*Vitamin B12	3.00	13.8	120	0.03
Platelets	7.83	11.6	180	>0.96
*LH	4.36	15.7	180	0.27
*FSH	3.43	19.7	180	0.27
TSH	4.62	8.88	300	0.99
Urate	6.67	13.6	300	>0.97
Cholesterol, HDL	1.67	7.00	300	0.16
ALP	5.42	13.8	450	0.99
Urea	4.26	11.1	450	0.91
Cholesterol, total	1.91	10.3	450	0.12
Triglyceride	2.63	32.5	600	0.19

Key Observations

• Insufficient numbers of patient samples limit the application of AoN algorithms for T3 uptake, CK, LD, amylase, total T4, lipase, total protein, prolactin, and vitamin B12, where less than 120 patient samples are available.
• Relatively high test volumes for LH and FSH are still too low because of the high s_pop/s_meas ratios, 15.7 and 19.7, resp.
• High test volumes like HDL cholesterol and total cholesterol are limited by the available process capability, as shown by the sigma-metrics of 1.91 and 1.67, resp.
• Triglyceride is a worst case scenario where there is a low 2.63 sigma process capability and a very high s_pop/s_meas ratio of 32.5. Even with 600 patient samples, the AoN algorithm is ineffective and provides a P_ed of only 0.19 or 19% chance of detecting medically important errors.

The available number of patient samples will limit the application of AoN algorithms in many laboratories. However, certain tests such as electrolytes may be monitored with relatively few patient samples. Large hospital laboratories, large screening programs, and references and commercial laboratories may make good use of AoN for many tests and analytical systems.

Applicability in a Specific Laboratory

The table below shows the error detection expected for the tests studied in a specific laboratory situation, i.e., relative to the sigma-metrics for the processes in a regional reference laboratory and the patient population being tested by that laboratory. This table shows the tests ordered top to bottom by the expected error detection, P_ed, highest at the top and lowest at the bottom.

Table 4. Expected error detection capability (P_ed) and the applicability of Average of Normals (AoN) patient data algorithms for each test and the number of patient specimens (subgroup n) in a regional reference laboratory.

Test Name	Sigma metric	spop/smeas	Subgroup n	Ped
AST	6.00	2.20	40	1.00
Chloride	3.57	2.58	30	1.00
Hematocrit	10.0	5.00	80	1.00
T4, free	3.43	2.52	100	0.99
Creatinine	5.56	4.44	40	0.99
Glucose	5.38	6.00	90	0.99
TSH	4.62	8.88	300	0.99
ALP	5.42	13.8	450	0.99
Potassium	3.33	2.78	100	0.98
ALT	6.18	5.37	40	>0.98
Urate	6.67	13.6	300	>0.97
Bicarbonate	4.35	3.48	40	0.96
Bilirubin, total	10.0	8.60	120	>0.96
Hemoglobin	10.0	9.09	120	>0.96
MCV	10.0	9.18	120	>0.96
RBC	10.0	9.71	120	>0.96
Platelets	7.83	11.6	180	>0.96
T3, total	4.80	2.18	10	0.94
Ferritin	4.00	7.89	120	0.94
Sodium	2.50	1.58	60	0.93
Phosphate	5.56	7.65	90	0.93
Calcium	2.86	1.84	60	0.91
Urea	4.26	11.1	450	0.91
LD	5.33	6.59	60	0.80
T3 uptake	4.80	3.00	20	0.76
Amylase	7.50	8.54	60	>0.63
Prolactin	3.43	6.33	120	0.56
CK	4.50	4.93	20	0.31
LH	4.36	15.7	180	0.27
FSH	3.43	19.7	180	0.27
Triglyceride	2.63	32.5	600	0.19
T4, total	4.80	6.10	60	0.18
Cholesterol, HDL	1.67	7.00	300	0.16
Lipase	10.0	12.7	60	>0.14
Cholesterol, total	1.91	10.3	450	0.12
Protein, total	2.63	4.94	80	0.09
Vitamin B12	3.00	13.8	120	0.03

Key Observations:

• About two-thirds of the tests in this laboratory could be monitored effectively with AoN algorithms, providing at least 90% detection of medically important errors. This takes into account the specific quality requirements, method performance characteristics, expected patient population, and actual number of patient samples available in this laboratory.
• These tests include methods having process capabilities as low as 2.50-sigma for sodium and 2.86-sigma for calcium.
• The fall off in error detection for the bottom third of the list is dramatic. The last ten tests will have expected P_eds of only 0.31 to 0.03 by AoN. This includes one test, Lipase, that has a process capability of 10.0 sigma, but a large s_pop/s_meas ratio of 12.7 and only 60 available patient samples. The next four tests would have expected P_eds of 0.56 to 0.80, including amylase whose process capability is 7.50, s_pop/s_meas ratio 8.54, and only 60 available patient samples.

Design of AoN algorithms

The proper design of AoN algorithms depends on the careful consideration of many factors and performance characteristics. The difficulty in assessing the impact of these many factors may very well be the main hurdle to implementing AoN algorithms today. The laboratory community sees the use of AoN as a way to save money, which means there is a strong driving force that should lead to more use of patient data to monitor process performance. But, what's the right AoN algorithm?

The preceding discussion shows the complexity of selecting or designing appropriate AoN algorithms in relation to critical factors such as process capability, the s_pop/s_meas ratio, and the number of patient test results. Practical design tools are especially needed for complicated applications like this and the OPSpecs chart can again serve this purpose. Computer support is also necessary to make the application simple.

For a demonstration of automatic selection of AoN rules see the EZ Rules homepage.

What's the point?

Properly designed AoN algorithms can address two major problems in analytical quality management in laboratories today:

• Provide adequate error detection for tests such as sodium and calcium where quality requirements are demanding and methods have relatively low process capability.
• Provide a measure of run length to determine when it is necessary to re-analyze controls and re-validate a process during continuous operation and production.
  Implementation should be part of a multi-stage QC procedure, where traditional statistical QC rules and materials are used to initially validate method performance and to monitor the process while the necessary number of patient samples are being accumulated. Once the necessary number of patient test results are available, the process monitoring can switch over to the AoN algorithm. Implementation of the data calculations can employ traditional averages, moving averages, or exponentially smoothed averages [5], which should be easily supported by the computer horsepower available in laboratories today.

References

1. The Use of Patient Data for Process Control: It's Time has Arrived. Cembrowski G. Clin Chem 2000;46(No. 6 Supplement):S18-S19.
2. Hoffman RG, Waid ME. The 'average of norlams' method of quality control. Am J Clin Pathol 1984;81:492-499.
3. Cembrowski GS, Chandler EP, Westgard JO. Assessment of "average of normals" quality control procedures and guidelines for implementation. Am J Clin Pathol 1984;81:492-497.
4. Westgard JO, Smith FA, Mountain PJ, Boss S. Design and assessment of average of normals (AON) patient data algorithms to maximize run lengths for automatic process control. Clin Chem 1996;42:1683-1688.
5. Smith FA, Kroft SH. Exponentially adjusted moving mean procedure for quality control: an optimized patient sample control procedure. Am J Clin Pathol 1996;105:44-51.