Tools, Technologies and Training for Healthcare Laboratories

QC: Not just a technicality!

November 2005

If we say reduced QC is actually equivalent QC, does it matter that this isn't true? In an era where telling the truth can be hazardous to your career, it's time to take stock of the state of truth here in the lab.

with Sten Westgard, BA, MS

Fair Warning: It's November in the US, so this essay discusses some political topics.

I continue to worry about the quality of healthcare and, in particular, the quality of laboratory tests. In the aftermath of the CLIA Final Rule and the CMS interpretative guidelines (as found in part C of the State Operations Manual, SOM), I have made it a point to speak out more vigorously against poor quality practices, such as the “Equivalent QC procedures” that have been proposed by CMS.

Equivalent QC sounds good. If you say it again and again and again – staying on message – people may even start to believe that it means what it says. Once again, at the November 2 AACC audio-conference on “CLIA and EQC Options for the Future,” CMS is repeating the same old message about EQC, in spite of the numerous objections that have been raised about the validity of the evaluation protocols, including objections from industry. At the recent QC for the Future conference, the industry group Advamed proposed the development of “Option 4” to provide a scientifically valid alternative QC strategy [1]:

“We were unable to understand the scientific reasoning behind the laboratories’ procedures for establishing EQC. For example, Option 1 allows the laboratory to perform 10 days of testing, and if all of the quality control functions properly, then they may switch to testing only once every 30 days. Traditional verification methods would require the laboratory to verify quality control functionality longer than 30 days before switching to the 30 day interval.”

“Some perjury technicality”

In late October, Senator Kay Bailey Hutchinson explained that the current CIA leak (sometimes known as Plamegate) investigation might lead to indictments based on “some perjury technicality.”

"Ms. Hutchison said she hoped 'that if there is going to be an indictment that says something happened, that it is an indictment on a crime and not some perjury technicality where they couldn't indict on the crime and so they go to something just to show that their two years of investigation was not a waste of time and taxpayer dollars.'"[2]

The Senator was correct about an indictment coming down (Scooter Libby, the Vice President’s chief of staff). Two days later, she backed off her statement, admitting, "I certainly think that if someone has lied to an investigator, of course that is a crime. It is a terrible crime."[3] Nevertheless, many pundits who once thought Bill Clinton's perjury was an impeachable offense are now saying that the perjury and obstruction of justice charges being made against the Bush administration are only minor crimes. 21st century politicians, apparently, take as a given that you aren't going to tell the truth all the time.

Unfortunately, politicians aren't the only group with a tenuous relationship with the truth. This year the SAT test began including an essay test. Unfortunately, telling the truth isn't part of that test. In the official guide for scoring, it explains:

"Writers may make errors in facts or information that do not affect the quality of their essays. For example, a writer may state 'The American Revolution began in 1842' or 'Anna Karenina', a play by the French author Joseph Conrad, was a very upbeat literary work.' ... You are scoring the writing, and not the correctness of the facts."[4]

[By the way, the correct answers are: 1775, Leo Tolstoy, and Anna throws herself under a train - whether that's upbeat or not depends on your taste in literature, I guess]

Worse still, an MIT professor, Dr.Les Perelman, analyzed the essays and scoring system of the new SAT, and found that there was a direct correlation between the length of the essay and its score. The longer your wrote, the higher your score, regardless of what you actually wrote. When asked how students should answer the essay question, Dr. Perelman said, "I would advise writing as long as possible and include lots of facts, even if they're made up."[4] While professor insisted, "It's exactly what we don't want to teach our kids." one can't help but note that this provides excellent preparation for going into politics.

Finally, in early November, we were treated to the spectacle of oil executives testifying before the US Senate Energy and Commerce committees. All the public outrage over high gas prices and even higher oil industry profits compelled the politicians to "do something," and that meant brow-beating the oil executives. But what was most interesting about the executives' testimony is what didn't happen: they were not sworn in (which is the norm) and their subsequent testimony was therefore not under oath. At one point in the Senate questioning, the executives of Exxon Mobil, ConocoPhillips, Shell Oil, and BP America denied that their companies had met with Vice-President Dick Cheney's energy task force back in 2001. As it turns out, however, those denials were false. A White House document obtained by the Washington Post shows that, in fact, Cheney's task force did meet with representatives from those oil companies. (Whether or not oil companies should have input in the United States energy policy is a contentious issue, which certainly won't be decided here in this paragraph.). But the biggest outcome of the hearing was that, having avoided testifying under oath, the oil executives also neatly avoided committing perjury. [5]

And there you have it: in politics, in academics, and in business, the truth is an endangered species.

Is truth still important in healthcare? I would argue that it is, because the truth is closely related to trust. And trust is critical if we are to depend on the actions and decisions of others in today’s complicated world. Can you depend on the healthcare system for critical care and assistance? Can you depend on your physician for the right advice and treatment? Can you depend on laboratory tests for making the correct decision on diagnosis and treatment? Can you depend on equivalent QC to guarantee the quality of laboratory tests?

Truth and quality

As I’ve said before, the standard for evidence in a courtroom – “do you swear to tell the truth, the whole truth, and nothing but the truth” – should also be the gold standard for evidence of quality [6]. To trust the quality of a laboratory test, we must know that it is correct within certain limits, that the imprecision and inaccuracy of the measurement procedure under stable operation will not exceed those limits, and that QC will detect unstable operation that would lead to errors that exceed those limits.

The three parts of the truth standard help us understand the importance of knowing the quality that is required, providing measurement procedures with the appropriate precision and accuracy, and implementing QC procedures that can detect medically important errors. All three dimensions are important if test results are to be trustworthy.

Equivalent QC

CLIA recognizes that the role of QC is to detect errors and provides at least 3 possible ways for doing so: (1) do the right QC to detect “immediate errors,” or (2) perform a minimum of 2 levels per day to detect “un-immediate errors”, or (3) possibly only 2 levels per week to detect “un-un-immediate” or even 2 levels per month to detect “un-un-un-immediate” errors. [Okay, I made up the un-immediate and un-un-immediate and un-un-un-immediate terms, but CMS really did make up the term “immediate error detection” and is also trying to make us believe that immediate means once a day or a week or even a month.]

That 3rd choice is called Equivalent QC, which actually provides 3 different options that depend on the extent or completeness of the built-in controls of the particular analytic system. However, there is nothing equivalent about EQC [6]. It is obvious that “immediate error detection” cannot be achieved by running controls once a day, once a week, or once a month. CMS and industry will argue that the application of EQC depends on the capability of internal, built-in, controls that will detect immediate errors, but we’ve pointed out in great detail on this website that the validation protocols recommended by CMS are NOT valid and do not guarantee the built-in controls actually work. You’re told to just trust what the manufacturer says about the factors and variables monitored by the built-in controls.

Equivalent QC is an insidious talking point because the name itself provided misinformation. Like the “No child left behind” program, the name “Equivalent QC” doesn’t necessarily describe the outcome. Truth in labeling should require that EQC be described as “Equivocal QC” or, better yet, “Reduced QC” (RQC).

Which is why, I suppose, CMS has proposed EQC. Under the pressure of doing more and more with less and less since CLIA, laboratories have finally “minimumed” out. Today, laboratories only have the resources to do the minimum needed to get by and are entering the stage of “doing less with less.” This means doing more tests with less quality with less personnel with less training. We have broken the back of laboratories and all those hard-working people who have given and given are ready to give up.

QC is still essential!

I still believe statistical QC is an essential tool for managing the quality of laboratory tests because laboratories need an independent and widely applicable tool. If laboratories are to maintain a responsibility for the quality of the test results they produce, statistical QC is still the best available tool. It is not a one size fits all tool! It can be adapted for the medical quality required for a test, the analytic performance achieved by a measuring procedure, and the sensitivity (or rejection characteristics) of the QC procedure itself.

“Finally, 2 key points to remember when implementing EQC: first, the decision on whether or not to implement EQC, and which EQC option to choose, is ultimately the responsibility of the laboratory director. Test manufacturers may provide recommendations and guidance, but the laboratory director is required to consider the recommendation in light of his or her individual laboratory’s unique situation…

“Second, laboratory professionals should remember that EQC is only l part of the comprehensive Quality Systems program needed to fulfill CLIA requirements and to provide good quality testing…

“The take home message is that it is more important for the laboratory director to ensure that all of the Quality Systems are working well and producing accurate test results than it is to debate over which or any of the EQC options to select.”

These final words come from Judy Yost, who is the Director of the Division of Laboratory Services at CMS [7]. In spite of what industry may be able to do to mitigate the risks of bad test results, the laboratory is still responsible to assure the quality of those test results. That’s why statistical QC is still so important - not just a technicality!

References

  1. Ochs, L. QC for the future: CLSI standard development and option 4 proposal. Lab Med 2005;36:639-40.
  2. Stevenson, Richard W, and Johnston, David, "Republicans Testing Ways to Blunt Leak Charges", New York Times, October 23, 2005.
  3. Fox News, Fox & Friends, October 25, 2005.
  4. Winerip, Michael, "SAT Essay Test Rewards Length and Ignores Errors", New York Times, May 4, 2005.
  5. Milbank, Dana and Blum, Justin, "Document Says Oil Chiefs Met With Cheney Task Force", Washington Post, November 16, 2005, page A01.
  6. Westgard JO, Darcy T. The truth about quality: medical usefulness and analytical reliability of laboratory tests. Clin Chim Acta 2004;346:3-11.
  7. Yost J. CLIA and equivalent quality control: Options for the future. Lab Med 2005;36:614-6.

James O. Westgard, PhD, is a professor of pathology and laboratory medicine at the University of Wisconsin Medical School, Madison. He also is president of Westgard QC, Inc., (Madison, Wis.) which provides tools, technology, and training for laboratory quality management.