Tools, Technologies and Training for Healthcare Laboratories

Truthful claims may not be enough

February 2007

This year will see the launch of a fleet of new regulations and standards for the laboratory and the manufacturer. But is it really enough to verify that an instrument does what the manufacturer says it does? Don't we need to assure that the instrument does something clinically useful, too?

Warning: this essay contains themes which touch on politics

In February, the US government prosecuted the trial of Lewis "Scooter" Libby, who was charged with perjury and obstruction of justice in the leak and disclosure of identity of an undercover CIA agent, Valerie Plame Wilson. Mr. Libby is the former chief-of-staff for Vice President Cheney, who declined to appear as a witness on Libby's behalf in the trial. While it is a federal offense to knowingly disclose the identity of a CIA agent, that violation of law was not the focus of the trial. This trial was about whether or not Libby told the truth to FBI officials and the Special Prosecutor who were investigating.

As we well know, truth is a scarce commodity in the real-world of politics. Politicians tend to tell a little bit of the truth, but seldom the whole truth and nothing but the truth. The problem for Libby is that he was under oath to tell the “truth, the whole truth, and nothing but the truth,” but his claims were widely disputed by several witnesses in the trial.

Truthful claims are also important in the laboratory world. “Truth in labeling” is the fundamental principle in the FDA’s review of new medical devices. In what is known as the “510k clearance process,” a manufacturer must submit the claims for performance, along with the experimental data that support those claims. Those claims must include precision, accuracy, linear range, reference range(s), etc. The FDA reviews the submitted data to be sure it supports the claims being made by the manufacturer, then “clears” the method for sale in the marketplace.

CLIA then takes over to regulate the use of the method in the marketplace. CLIA focuses on the management of quality in the real-world of individual laboratories. Part of the CLIA regulation, section 493.1253, is a standard for “establishment and verification of performance claims.” For any new method or test system introduced after April 24, 2003, laboratories are required to demonstrate that they “can obtain performance specifications comparable to those established by the manufacturer.”

Establishment vs verification of claims

The establishment of claims typically requires more data than verification of those claims. For example, to establish performance claims, manufacturers will typically test a new analytic system in two or more laboratories and will collect much more data than what is practical in method validation studies in routine service laboratories. It makes sense that a service laboratory should compare the performance it observes in smaller studies to that claimed by the manufacturer.

This is sometimes done subjectively by looking at the claim and the lab’s estimate of performance, then judging whether the numbers are close enough. This is a likely outcome when following the directions in CMS Brochure #2 “Verification of Performance Specifications” because it provides no guidelines for the analysis of the method validation data, only stating that:

“Once the laboratory director has reviewed and approved the results of the verification studies, the laboratory may begin using the test system for routine testing and reporting patient test results.”

A more objective determination can be made with the aid of statistical “tests of significance,” where the manufacturer’s claim is compared with the observed performance to determine whether or not there is a “statistically significant” difference. “Statistically significant” simply indicates whether difference observed is “real,” i.e., whether or not the data is sufficient to demonstrate a difference between the claim and the performance observed in the laboratory. It is possible there is a large difference, but not enough data to demonstrate it is real, or alternatively, there may be a small difference that can be demonstrated to be real because a lot of data has been collected. The amount of data available has a major influence on whether or not a difference can be demonstrated.

Statistical significance vs clinical significance

There is one major fault with this approach. Manufacturers do not make any claim that a method or test system provides the quality needed for medical application of the test results, i.e., FDA clearance does not require a claim for quality! To do so, a manufacturer would need to state a quality specification, e.g., the allowable total error, the maximum allowable SD, or the maximum allowable bias, then demonstrate that the new method or test system has less error than specified by those allowable limits of error.

For example, note the guidelines for “analytical quality” in the instructions for authors for Clinical Chemistry.

“Results obtained for the performance characteristics should be compared objectively with well-documented quality specifications, e.g., published data on the state of the art, performance required by regulatory bodies such as CLIA-88, or recommendations documented by expert professional groups.”

It is also interesting to note that CMS, in Appendix C of the State Operations Manual, provides the following interpretive guideline [D5421 regarding §493.1253(b)(1)]:

“The laboratory is responsible for verifying the performance specifications of each nonwaived unmodified FDA-cleared or approved test system that it introduces prior to reporting patient test results. The verification of method performance should provide evidence that the accuracy, precision, and reportable range of the procedure are adequate to meet the clients’ needs, as determined by the laboratory director and clinical consultant. A laboratory may use the manufacturer’s performance specifications as a guideline, but is responsible for verifying the manufacturer’s analytical claims before initiating patient testing.”

The phrase “adequate to meet the clients’ needs” implies that the method performance characteristics be compared to the quality of test results needed for clinical services. The phrase “as determined by the laboratory director and clinical consultant” implies that this is a professional responsibility of the laboratory and its directors.

What’s the point?

There’s a big difference between (a) verifying a manufacturer’s claims for precision and accuracy vs (b) assessing whether a manufacturer’s method or test system provides clinically or medically useful test results. Verification of claims considers only whether your laboratory performance is consistent with that claimed by the manufacturer. Assessment of clinical or medical significance considers whether your laboratory performance satisfies the quality specification defined for the test.

In short, the difference between these two approaches can be likened to the difference between the “statistical significance” vs “clinical significance” of test results. It is possible that the performance of new method or test system is consistent with the manufacturer’s claims, but still NOT be acceptable for the clinical and medical applications of your clients. As a laboratory professional, you need to be sure that the test results satisfy the needs of your clients.


James O. Westgard, PhD, is a professor emritus of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.