July 1999

An updated version of this essay appears in the Nothing but the Truth about Quality manual

James O. Westgard, PhD, FACB

Efforts to provide worldwide standards have been led by the International Organization of Standards, or ISO as it is commonly known. The ISO standards were initially aimed at industry but are now being adapted to healthcare laboratories. One of the areas currently under discussion is the standardization of concepts, definitions, and terminology related to analytical quality management, as discussed by Dr. Xavier Fuentes-Arderiu in an essay on "Trueness and Uncertainty" and an associated "Glossary of ISO Metrological and Related Terms and Definitions Relevant to Clinical Laboratory Sciences."

I applaud efforts to standardize and clarify concepts, definitions, and terminology, but I believe there are serious issues that need to be resolved before the proposed ISO definitions and terminology are adopted.

Evolving concepts of analytical quality

Precision and accuracy were the terms being used to describe the performance of analytical measurements in the 1960s when healthcare laboratories began to be automated and to grow into high volume production operations. In contrast to many industrial applications where replicate test measurements are made routinely, single test measurements are the norm in healthcare testing. This distinction is important because the effects of imprecision can be minimized by performing multiple measurements, hence industrial applications could easily control the amount of random analytical error by increasing the number of replicate measurements. Therefore, the most important measure of industrial quality was the inaccuracy or systematic error.

Total analytic error was recommended in the mid 70s [1] as a better way to evaluate the performance of clinical measurements where only a single measurement is generally made in determining a test result. All individual test results may be in error due to both the imprecision and inaccuracy of the method, therefore the combination of the two errors, or the total analytical error, determines the quality of the test result. In developing this new concept, the emphasis was on "errors", their random and systematic components, and the net or total effect of those components. It took over a decade for this concept to become established in healthcare laboratories. Its relevance is especially clear in the US today because the CLIA laboratory regulations specify criteria for acceptable performance in the form of allowable total errors that are used in grading analytical performance in proficiency testing surveys [2].

Operating specifications were developed in the early 90s as an outgrowth of total quality management and the interest in making quality control a quantitative technique for managing routine production [3]. The operating specifications for a method define the imprecision and inaccuracy that are allowable and the QC that is necessary (control rules, number of control measurements) to assure that a stated quality requirement will be achieved in routine production. The important point here is that operating specifications consider both the stable performance of the method (imprecision and inaccuracy) and the capability of the QC procedure to detect changes (unstable performance).

ISO concepts of trueness and uncertainty

Trueness is used by ISO to describe the "closeness of agreement between the mean obtained from a large series of results of measurement and a true value." The emphasis on the "mean obtained from a large series of results of measurement" limits this concept to the systematic error or inaccuracy (bias) of a method.

Uncertainty is used by ISO to describe a "parameter, associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand." This term could be quantitatively described by calculating a standard deviation or some multiple (confidence interval).

In short, the new ISO terminology recommends trueness instead of accuracy, inaccuracy, and systematic error and uncertainty instead of random error, imprecision, and precision. Total error wouldn't exist! Furthermore, estimates of uncertainty would combine different sources and estimates of random error through propagation of errors calculations. These calculations would require a competent statistician, a clinical chemist with some mathematical or statistical aptitude, or special computer programs that could be used by laboratory analysts.

The objective is to describe the uncertainty of a measurement or test result. For example, a test result of 100 means a value in the range from say 96 to 104. The test result is good to within 4 units. Of course, that also means it may be off by up to 4 units, but the result isn't in error - just uncertain!

Difficulties in implementing the ISO recommendation

Any change is difficult - even one that is entirely beneficial. A change becomes impossible when the difficulties outweigh the benefits. One way to illustrate the benefits versus difficulties is to use a "force field diagram" to describe the forces that are driving the change vs those that are restraining the change. The accompanying diagram identifies the driving forces as being the benefit (in principle) of uniform concepts and terminology for communication in an international marketplace, benefit of uniformity with other industries, the ISO momentum in the European marketplace, and the benefit from a pure theory of measurements. The restraining forces represent the following difficulties:

• Need for statistical skills. Few healthcare practitioners, whether laboratorians, nurses, or physicians, are skilled in statistics. Their training is quite different from the strong mathematical orientation of the engineers who are in key positions in other industries. Healthcare personnel often find statistics to be confusing. They certainly don't have a quantitative understanding of uncertainty, however, they do have a quantitative understanding of errors. They know that errors are bad, should be eliminated when possible, and always kept small relative to the medical use and interpretation of test results. The concept of error has real meaning whereas the concept of uncertainty has only statistical meaning.
• Trend towards lower skilled personnel. The application of more complicated concepts usually requires more sophisticated practitioners. Today, the education, training, and skills of professionals in healthcare laboratories are actually decreasing, as part of the efforts to reduce the costs of healthcare. There will likely be fewer PhD clinical chemists available to deal with the complicated calculations that are necessary to properly estimate the uncertainty of test results. The availability of computer programs may facilitate the calculations, but may not improve the understanding and interpretation of the calculated results.
• No practical benefits over the error concept and terminology. There are no practical benefits to practitioners to change from the understandable concepts and terminology of errors to more difficult statistical concepts and terminology. The error concepts do not in any way limit current practices in quality management. The real problem is that laboratories need to define the quality required for the tests they provide.
• Quality requirements already defined as allowable errors. Given that quality requirements must be defined if analytical quality management is to be improved, it is important to recognize that most recommendations for analytical quality are given in the form of allowable errors - allowable SD, allowable bias, allowable total error. What amount of uncertainty is allowable? How do you develop recommendations for the allowable uncertainty?
• Changes will take a long time. In the late 1970s, IFCC recommended that the terms imprecision and inaccuracy replace precision and accuracy to properly emphasize that the deviations (or errors) from the true or correct values. Several years went by before the new terms of imprecision and inaccuracy were being used in the scientific literature. Likewise, the concept of total error took about ten years from introduction to general acceptance by the laboratory community. Operating specifications, which were first introduced in 1990, are not yet universally accepted. It can be expected that the new ISO concepts of trueness and uncertainty will take a decade or more to become established and useful.

And the answer is "roles for both"

The practical management of analytical quality in a service laboratory is best served by concepts and terminology that relate to analytical errors, rather than by the newly recommended concepts of trueness and uncertainty. Current quality requirements and quality management tools are based on error concepts and theory.

The role for the new concepts is in the applications of reference methods, not field methods. Reference methods are used to establish true values or to assign values to calibration materials. In such applications, it is important to make replicate measurements and minimize random errors, to correct for known systematic errors, and then to be able to express the uncertainty in the measured or assigned values.

References:

1. Westgard JO, Carey RN, Wold S. Criteria for judging precision and accuracy in method development and evaluation. Clin Chem 1974;20:825-33.
2. Westgard JO, Wiebe DA. Cholesterol operational process specifications for assuring the quality required by CLIA proficiency testing. Clin Chem 1991;37:1938-44.
3. Westgard JO. Charts of operational process specifications ("OPSpecs charts") for assessing the precision, accuracy, and quality control needed to satisfy proficiency testing criteria. Clin Chem 1992;38:1226-33.

Other Essays:

Myths of Quality Putting Quality into Quality Control Assuring Quality through Total Quality Management Trends in quality management: Utilization and Outcomes Quality Goals, Requirements, & Specifications Future Directions in Quality Control The Myth of Medical Decision Limits Quality by Design Tools and Technology for QC Training Education and Training for Analytical Quality Management, Part I Mapping the Road to Analytical Quality with OPSpecs Charts Quality and Commerce QC - Back to Basics Education and Training for Analytical Quality Management, Part II: Developing Web-courses Method Validation - The Inner, Hidden, Deeper, Secret Meaning Education and Training in Analytical Quality Management, Part III: Basic QC Training Electronic QC and the Total Testing Process From Rules and Tools to Technology and Training (Beijing) Quality Requirements: the debate heats up Z-Stats: A treat and a treatment The Need for a System of Quality Standards What's wrong with traditional QC? To be Uncertain or In Error? That is the Question QC 2000 Education and Training for Analytical Quality Management, Part IV: Interactive Training Tools Do's and Dont's of QC The Abbott Consent Decree: A Wake-Up Call WQC Y2K Sage Advice about new approaches to Quality Control EZ Rules for Assuring Quality Who will care to quality tomorrow? Quality is Job 1 when the rubber meets the road Errors in reasoning about Laboratory errors Six Sigma Quality Management & Lab Precision Six Sigma Quality Managment & Requisite Lab QC 2001: Year of the Odyssey essays CLIA Postponed again and again and again Repeated, Repeated, Got Lucky Six Sigma Staffing Strategies Technology for Implementing QC Right $aving the Cost$ of Poor Quality

Has Quality been "Enronned"? Why not Evidence-Based Method Specifications? Quality: "I think I got it!" Cooking the Books: Does it happen in the Lab? CLIA QC Clearance - A momentous happening Signs of Six Sigma Good Data Wanted, Bad Data Need Not Apply Final, final, final, final, final CLIA Rules The Truth Standard for Quality It's an Honor: Reflections on being a Teacher 2004 JCAHO Patient Safety Goals ISO Says So Medical Errors: Complexity and Its Solutions Giving Thanks for 2003: Observations on the state of Quality Autoverification: Taking QC to the next level - is that up or down? Think straight, Talk straight The Gospel According to ISO More on Eqc and "Quality-Less" Compliance Testing Equivalent Quality: A better way The Final Word on the Final Rule? Hear, Hear, Hearings on Untruth and Unquality, Part I Hearings on Untruth, Part II: Cracks Hearings on Untruth, Part III: Facts Hearings on Untruth, Part III: Broken Windows Connecting the Dots Hearings on Untruth, Part IV: Inadequate Inspections Hearings on Untruth, Part V: Bad Apples or Tip of the Iceberg? The Quality of Laboratory Testing, Part I No Laboratory Left Behind Vioxx and Values, Vaccines and Votes The Quality of Laboratory Testing: Methodology The Quality of Cholesterol Testing Bah, Humbug! How I learned to love EQC The Quality of Glucose Testing The Quality of Calcium Testing Blowing the Whistle on the Tip of the Iceberg The Quality of Glycohemoglobin The Quality of PSA Testing Solutions for the Taxing Problem of QC The quality of Coagulation Testing The variability of estimates from PT surveys Links to India, Part I Test Quality vs. Method Performance QC: Not just a technicality 2005 in Review: 100,000 miles to Quality Unannounced Inspections, Unknown Consequences Hopeful Healthcare in a Fearful Society Quality Indicators and Benchmarks Trouble with Tracking Tests