February 2007

We need some truthful guidance about Trueness, Uncertainty and Quality. This year's regulations feature new terminology that adheres to ISO conventions. As we leave behind our familiar concepts of precision, bias, and total error, are we gaining anything? Are we just changing names, or are we changing our goals, too?

• Trueness and uncertainty
• Insights about "truth"
• Insights about "reasonable doubt"
• Implications for performance and quality characteristics
• Implications for performance and quality requirements
• "Truthiness" vs Truthfulness
• References

Warning: this essay contains themes which touch on politics

The trial of "Scooter" Libby on charges of perjury is in its final weeks as I write this essay. Libby, who was chief-of-staff for Vice President Cheney, is accused of lying to the Grand Jury that was investigating the outing of a CIA agent. At this point in the trial, the evidence has been presented, the closing arguments completed, and the case has gone to the jury. The issue of interest now is “reasonable doubt,” which we might understand as the evaluation of the truth and uncertainty of the evidence, upon which a judgment or decision will be made.

In our own world of laboratory testing, it is also important to evaluate the truth and uncertainty of test results because they provide evidence that leads to judgments about the diagnosis and treatment of patients. We may hope that laboratory test results represent the truth, but we should really do more – we should guarantee that the quality of our test results satisfies clinical needs. Otherwise we do injustice to our patients.

Thinking about the judicial process, particularly the concepts of truth and reasonable doubt, reveals some close parallels to the laboratory concepts of trueness and uncertainty, as well as some interesting insights into the meaning of quality.

Trueness and uncertainty

In the new terminology for describing the quality of a laboratory testing, trueness replaces the bias concept of accuracy (or inaccuracy) and uncertainty replaces the total error concept of accuracy. This change in terminology is being driven by ISO (International Standards Organization), particularly the ISO 15189 document which describes particular requirements for quality and competence in medical laboratories [1].

The ISO terminology is being adopted by CLSI (Clinical Laboratory Standards Institute), formerly known as NCCLS (National Committee for Clinical Laboratory Standards). For example, the recent revision of the EP15-A2 document is now titled “User Verification of Performance for Precision and Trueness” [2]. CLSI also has committees working on guidelines for estimating uncertainty. The consequence for US laboratories is that they will have to adopt the new terminology, whether or not they adopt ISO 15189.

Unfortunately, these new terms and the related concepts fall short of providing a proper definition and characterization of the quality of a laboratory test for healthcare applications. ISO has force-fit the terms and concepts from metrology laboratories without sufficient consideration for certain unique characteristics of healthcare laboratories, such as the following:

• the practice of making only a single measurement on a clinical specimen for healthcare applications, rather than the common practice of making replicate measurements in metrology laboratories;
• the low level of quality control performed in healthcare laboratories, which often does not provide sufficient detection of medically important errors;
• the expected variability due to the specimen itself, particularly the intra-individual biologic variation of human subjects; and
• the existence of error models that have been developed and tailored to fit these characteristics of healthcare laboratories.

Insights about “truth”

Recalling the courtroom process, evidence must be submitted under oath to assure its truth. When applied to the laboratory, the age old standard for truth implies that a laboratory test should “tell the truth, the whole truth, and nothing but the truth.” This truth standard can help us understand the important dimensions of quality, the relationship between trueness and uncertainty, and the additional characteristics that are important to assure the quality of a test result. In our laboratory application, think about the three dimensions of the truth standard in the following way:

• Truth relates to the bias observed for a measurement procedure;
• The whole truth relates to the analytical uncertainty of a measurement procedure, which is the combined effect of its bias and imprecision of a measurement; under the Guide for estimating the Uncertainty of Measurements [GUM, 2-3] , uncertainty can be expressed in the form of a standard deviation, or a multiple of the standard deviation corresponding to 95% or 99.7% limits for “expanded uncertainty;”
• Nothing but the truth relates to the analytical quality of a testing process, taking into account the additional uncertainty due to Quality Control (lack of sensitivity for detecting unstable operation); it also relates to the clinical quality of a test result by accounting for the uncertainty due to any variability in the sample, including a patient’s own biologic variability.

Insights about “reasonable doubt”

ISO recognizes there are other considerations in addition to trueness and uncertainty with its guidance for the design of QC procedures, as found in section 5.6.1 of ISO 15189 [1]:

“The laboratory shall design internal quality control procedures that verify the attainment of the intended quality of results…”

This guideline addresses the reasonable doubt about a lab test and the need provide proper quality control to guarantee the desired quality is attained.

The “design internal quality control procedures” must take into account their “power” to detect errors, which depends on the control rules used and number of control measurements in a run. Therefore, the analytical quality of a test involves more than just trueness and uncertainty of the measurement procedure. It also involves the QC procedure that is necessary to verify or assure that the intended quality is actually achieved by an analytical testing process.

“Intended quality of results” acknowledges that the clinical use of the test must be considered in designing the QC procedure. Clinical use may be best understood by describing the test results that lead to different clinical actions and should consider pre-analytic factors that may contribute to variability, including the patient’s own biologic variability. Otherwise, intra-individual biologic variation contributes to the reasonable doubt that a laboratory test provides reliable evidence.

Implications for performance and quality characteristics

In the context that truth must be established beyond reasonable doubt, we can see that there are other considerations than just trueness and uncertainty. The clinical quality of a test result is affected by the intra-individual biologic variation, the sensitivity of the QC procedure to detect clinically important errors, the random error (precision, imprecision) of the measurement procedure, and the systematic error (accuracy, inaccuracy) relative to the true or correct value. In this framework, the clinical quality of a test result encompasses the analytical quality of the testing process, which in turn encompasses the analytical uncertainty of the measurement procedure, as well as its trueness. The ISO concepts of trueness and uncertainty by themselves fall short because they do not consider the important effects of intra-individual biologic variation and quality control, which must be accounted for to minimize, or better yet, eliminate the “reasonable doubt” of a laboratory test result.

These insights lead to the following hierarchy:

• Clinical quality of a test result
• Analytical quality of a testing process
• Uncertainty of the measurement procedure
• Trueness of the measurement procedure

Trueness and uncertainty are performance characteristics, not quality characteristics. To manage the analytical quality of a laboratory testing process, the laboratory must take into account the QC that is needed to guarantee or assure the attainment of the desired quality; to manage clinical quality, the laboratory must, in addition, take into account the intra-individual biologic variability (and possible other pre-analytic factors) that will affect a test result.

Implications for quality and performance requirements

Given the hierarchy of quality and performance characteristics, there is a need for a similar hierarchy of quality and performance requirements, which are called quality specifications in today’s terminology. In my way of thinking, the term quality specifications blurs the line between performance and quality, which I consider to be different characteristics, as argued above. I would prefer to call these quality and performance requirements, or goals, because they all represent the maximum allowable change, error, uncertainty, SD, or bias, and all need to be translated into bench-level “operating specifications” for practical application in the laboratory [5]. Nonetheless, they are lumped together today, amidst a lot of confusion about their meaning and application.

Let me suggest the following hierarchy for quality and performance requirements:

• A requirement for clinical quality may be stated in the form of a clinical decision interval, i.e., the gray zone between the test results that lead to different decisions for classification or treatment, which should encompass intra-individual biologic variation, the QC safety margin, and the measurement uncertainty, which is a function of the bias and imprecision of the measurement procedure (its trueness and uncertainty).
• A requirement for analytical quality may be stated in the form of an allowable total error, which should be interpreted as encompassing the safety margin for the QC procedure and the measurement uncertainty (or expanded uncertainty), which is a function of its trueness (bias) and imprecision.
• A specification for measurement uncertainty may be stated in the form of the maximum allowable bias and the maximum allowable imprecision, which can then be combined by an appropriate model to define the maximum allowable expanded uncertainty.
• A specification for measurement trueness should be stated in the form of a maximum allowable bias.

For applications in healthcare laboratories, it should be preferred to define clinical or analytical quality requirements when the purpose is to validate a new method and manage its quality in routine operation. Specifications for measurement trueness, uncertainty, and expanded uncertainty are needed mainly for purposes of certification or accreditation.

"Truthiness" vs Truthfulness

The term “truthiness” has been used to characterize US politics today and might apply to US healthcare quality as well. Truthiness has to do with the appearance of truth, not the actual factual truth of the matter. Quality today also has much to do with appearance rather than fact. In healthcare, we have implemented so many quality programs and applied so many quality tools that somehow, magically, we believe quality is being achieved. How can we achieve quality if we don’t understand what it is and haven’t even defined any quantitative requirements for how good we need to be? In the laboratory, how can we achieve quality if we don’t understand trueness and uncertainty and how they differ from the analytical quality of a testing process and the clinical quality of a test result? How can we verify the attainment of the intended quality of test results if we don’t even now what quality is necessary?

The jury is still out!

References

1. ISO/FDIS 15189 Medical laboratories – Particular requirements for quality and competence. 2002
2. CLSI EP15-A2. User Verification of Performance for Precision and Trueness. Clinical Laboratory Standards Institute, Wayne PA, 2005. (Member cost $60, nonmember cost $120).
3. GUM. Guide to the expression of uncertainty in measurement. ISO, Geneva, 1993.
4. Quantifying Uncertainty in Analytical Measurement – 2nd edition (QUAM:2000.P1). EURACHEM/Co-operation on international traceability in analytical chemistry. www.eurachem.ul.pt/guides/QUAM3000-1.pdf
5. Westgard JO. The need for a system of quality standards for modern quality management. Scand J Clin Lab Invest 1999;59:483-486.

James O. Westgard, PhD, is a professor emritus of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.