What if the Frequency of Changing QC is Zero?

We know how to change QC frequency in a data-driven, evidence-based, patient-risk focused way. But we don't seem to be doing it at all.

What if the frequency of Changing QC Frequency is Zero?

Sten Westgard, MS
October 2020

It’s been 12 years since the landmark paper, Assessing the impact of the frequency of quality control testing on the quality of reported patient results [1],  by Dr. Curt Parvin, was published in Clinical Chemistry. It has not been fully appreciated as the groundbreaking innovation that is truly is. Before that paper, all QC frequencies were arbitrary. Afterward, it was possible to scientifically determine how often you needed to run QC. A question that had been without solution for decades was finally answered: How often should I run my QC? 

We can calculate that for you.

In the dozen years since its publication, further incremental improvements in the approach have been made to make it more practical to implement. Commercial software implementation was made available.[Bio Rad Mission Control]. A simple graphic guide was published [Westgard Sigma Rules with QC Frequency].  Yago and Alcover made a significant leap forward when they converted the mathematics into a simple graphic tool [2] for single control rules, followed by Hassan Bayat’s graphical tool for multiple control rules.[3] While these graphical tools were becoming available,  a new CLSI C24[4] guideline was published that described a “roadmap” for planning risk-based SQC procedures, but unfortunately didn’t provide any specific tools. In the literature, this roadmap has been adopted together with graphical tools to demonstrate applications  for single tests[5] as well as continuous production automated systems.[6]

There’s just one problem: it seems like no one is doing it. No one is changing their QC frequency. We might have been worrying for decades about how to solve this problem, but the solution doesn’t match up with the reality of running a laboratory in 2020.

In a handful of publications, a few laboratories have applied the Parvin approach to their tests, with results that are mixed at best, and wildly impractical at worst.

A handful of attempts at implementation

If the rule of thumb is that any new innovation takes a decade from theory to practice, QC frequency is starting to look like a laggard.

Few papers in the scientific literature have embraced the Parvin approach, and those that have may only half-heartedly implemented it.

Let’s take a look at a few publications.

The application of Six Sigma to perform quality analyses of plasma proteins
Fumeng Yang , Wenjun Wang , Qian Liu , Xizhen Wang , Guangrong Bian , Shijie Teng , Wei Liang Ann Clin Biochem 2020 Mar;57(2):121-127. DOI: 10.1177/0004563219892023

In this study, IgA, IgG, IgM, C3, C4, CRP, and RF were assessed on an IMMAGE 800 automatic analyzer using Chinese national goals from the National Center for Clinical Laboratories (NCCL), Rilibak goals from German, and goals derived from biological variations. Using the NCCL goals as their benchmark, they assessed Sigma metrics and used the simplified Westgard Sigma Rules with QC Frequency recommendations.

Key here to notice is that using biological variation derived performance specifications, several  assays would be considered terrible, having sigmas of 3.5 or less, and would require full “Westgard Rules” and a high  frequency of controls, with run sizes as small as  10 samples. The authors concluded that the biological goals were too stringent for today's methods.

In contrast, using the NCCL goals, few “Westgard Rules” are needed, and QC frequency is between every 450 patient samples to 1000 patient samples.  Depending on the actual workload, it may be possible to reduce the number of controls and the stringency of the rules to match the run sizes to the laboratory workload. 

Evaluation of the analytical performance of endocrine analytes using sigma metrics Yanming Liu, Yue Cao, Xijun Liu, Liangyin Wu, Wencan Cai Journal of Clinical Laboratory Analysis, 2020. Published online 20 September 2020 https://doi.org/10.1002/jcla.23581

This is perhaps the most comprehensive analysis performed on sigma metrics and QC Frequency. It focuses on 13 analytes, FT3, TT3, FT4, TT4, TSH, Cortisol, Estradiol, FSH, LH, progesterone, prolactin, testosterone, and insulin, on the Roche analyzer E602 using NCCL and EFLM goals (the latest iteration of biological variation derived performance specifications). Bias was determined through the national EQA program of NCCL. Precision was determined on Roche supplied controls, a weakness of the study.

In every case, the actual volume of the laboratory was a fraction the possible QC frequency, or run size.   The SQC design should consider the large workload for FT4 and the relatively low sigma to establish a maximum design, then consider the low volume tests to set another design.  Using the Sigma Run Size Nomogram, a 1:3s/2:2s/R:4s/4:1s/6x multirule with N=6 would be appropriate for the FT4 (and also Insulin), whereas a 1:3s/2:2s/R:4s with N=2 would be appropriate for relatively low workload of most other tests.    [The sigma metrics using EFLM goals were mostly dismal.]

Practical application of the sigma-metric run size nomogram for multistage bracketed statistical quality control analysis of eight enzymes Yuping Zeng, He He, Ken Qin, Mei Zhang, Zhenmei An, Hengjian Huang Clin Chem Acta 2019 May;492:57-61. DOI: 10.1016/j.cca.2019.02.006

This was one of the earliest studies that tried to implement the QC frequency. They looked at a Roche cobas 8000 c702 using Bio-Rad controls to assess imprecision, and using the national EQA program to determine bias. This study went even further, implementing a “start up” and a “monitor” design for QC.

In this group of tests, ALP has the lowest sigma and requires the most stringent control.  Most of the other tests  provide approximately 5 sigma performance or higher, which means a 1:3s/2:2s/R:4s/4:1s multirule with N=4 would be appropriate for a startup design and a large workload  of 500 samples or more.  Depending on the relative risk for ALP, we might assume a higher risk here (MaxENuf =2), in which case the run size could be doubled to make this test fit more generally with the run sizes for the others.    The practicality of the design for general implementation will often require adjustments between theory and practices, i.e., approximating the run sizes and systematizing the rules and Ns that are used.

Quality planning and control strategy for AQT90 flex Radiometer ® in point of care testing Claudio Ilardo , Cecile Reynaud , Regine Bonneton , Joel Barthes Scand J Clin Lab Invest 2020 Sep;80(5):427-432. DOI: 10.1080/00365513.2020.1768585

In this case, the authors focused on a blood gas instrument, the AQT90 flex, using the manufacturer’s controls to determine imprecision (a weakness of the study), and EQA by Probioqual to determine bias. Small blood gas devices are a particular challenge, given that each device will run a highly variable number of tests on any given day, but overall these are usually a low volume devices. They used a variety of TEa goals, some from the biological variation specifications (in this case Ricos, not EFLM), others were state of the art goals.

Obviously, that’s a huge variability in QC frequency, with some cardiac markers requiring essentially constant QC, and other assays need 200 times less QC.

To wrap this up, we’ve seen attempts to apply the Parvin approach to a wide array of testing, but few practical applications have emerged.

What obstacles are preventing the implementation of customized QC frequency?

The multiconstituent maximization

With most controls covering multiple assays, we’re not able to customize each assay’s frequency. If your control covers 40 or more analytes, you are locking yourself into one QC frequency for all of them.  If you have a variety of QC frequencies suggested by Sigma-metric analysis, and as the above examples show, there’s a big range of possible frequencies for analytes in the same category, how will you design QC? Will you run the QC as frequently as necessary for the worst analyte (which means you will be running too often for all the other analytes), or vice versa (running less than recommended for a few analytes). The cost efficiencies of having one multiconstituent control cover many tests are going to dominate the efficiencies that could be possible from optimized QC frequency.  You will have to make approximations related to the minimum sigma, or average sigmas for the group of tests.  You may also consider the relative risk for the different tests.  Doubling the risk means the run size can be doubled for that particular test.  You should also build your QC on a family of rules so those rules can be varied to match the error detection wanted.   

The compliance complex

Laboratories are ever obsessed with passing inspections, meeting minimum accreditation requirements, doing the least possible to meet regulations. While laboratories are encouraged to go beyond the minimum requirements, only a very few ever will. And until the regulatory and accreditation bodies add language that advises or requires better QC frequency decisions, those labs seeking to “get by” will feel no compulsion to do so.

The anxiety avoidance

Laboratory managers get used to routines and habits. Change, while constant in a laboratory, is instinctively avoided when possible. If everything else is changing, having a few things that stay the same is comforting. One of the few constants in the laboratory has been, up until now, a daily QC routine, and a fixed rule that’s easy to follow. The idea that every test might have its own individual QC frequency is unsettling. Many laboratories, dealing with so many other upheavals, will avoid the QC frequency challenge just as a way to keep their stress low. Psychology, as it so often happens, sometimes trumps all the mathematics and science in the world.

The debate on disparate specification delay

A final challenge to implementing an optimized QC frequency lies a bit upstream – the choice of performance specifications that drive the sigma metrics (which in turn drive the QC frequency). As many are aware, there is no single set of performance specifications. CLIA has on set of goals, the Australasian RCPA has another, the German Rilibak has yet another, and the European Federation of Laboratory Medicine has still another set of biological variation derived performance specifications. There are ongoing debates about which goals are better, or more appropriate, or more practical. And while that argument continues, it may feel safer to avoid the downstream implications of a particular sigma metric. Unfortunately, however, we should expect an perpetual debate on the best performance specifications, as new technologies are introduced, as the clinical use of test results evolve and are refined, and as new tests are added. Apathy in the face of

Conclusion

QC frequency is changing, but there are still further changes that will be necessary to make the optimization of QC practical. The ultimate optimization may only occur when QC is more internalized and automated, in a fashion that allows the individual scheduling of test QC.

In the short run, the current state of QC frequency tools will reward the high volume laboratories with the necessary skills to assess and manage the sigma metrics of their methods.

References

1. Parvin CA. Assessing the impact of the frequency of quality control testing on the quality of reported patient results. Clin Chem 2008;4(12):2049-54. DOI:10.1373/clinchem.2008.113639
2. Selecting Statistical Procedures for Quality Control Planning Based on Risk Management Martín Yago, Silvia Alcover  Clinical Chemistry, Volume 62, Issue 7, 1 July 2016, Pages 959–965, https://doi.org/10.1373/clinchem.2015.254094
3. Bayat. Selecting multi-rule quality control procedures based on patient risk. Clin Chem Lab Med. 2017 Oct 26;55(11):1702-1708. DOI: 10.1515/cclm-2016-1077
4. Bayat, Westgard, & Westgard. Planning Risk-Based Statistical Quality Control Strategies: Graphical Tools to Support the New Clinical and Laboratory Standards Institute C24-Ed4 Guidance Hassan Bayat, Sten A Westgard, James O Westgard The Journal of Applied Laboratory Medicine, Volume 2, Issue 2, 1 September 2017, Pages 211–221. https://doi.org/10.1373/jalm.2017.023192
5. Westgard, Bayat & Westgard. Selecting a Risk-Based SQC Procedure for a HbA1c Total QC Plan.  J Diabetes Sci Technol. 2018 Jul;12(4):780-785. DOI: 10.1177/1932296817729488
6. Westgard, Hassan & Westgard. Planning Risk-Based SQC Schedules for Bracketed Operation of Continuous Production Analyzers Clinical Chemistry Feb 2018, 64 (2) 289-296; DOI: 10.1373/clinchem.2017.278291