Quality Requirements and Standards

In this, the final installment of an extended discussion of HbA1c methods and the analytical quality necessary for patient care, we hear feedback from the original authors of the 2010 study.

Erna Lenters, PhD

April 2010

• Background
• Comments on Real World Learning Episodes
  
• References

This exercise is the conclusion of the RWL 2010 example of method validation studies for HbA1c.

• Episode 1 identified an abstract from the 2009 AACC national meeting [1], a validation study published in the January 2010 issue of Clinical Chemistry [2], and an accompanying editorial in that same issue [3].
• Episode 2 discussed the information available in the abstract and concluded that any decision on acceptable performance depended on knowing the quality needed for the clinical use of a HbA1c test. 
• Episode 3 reviewed the recommendations for quality requirements, as discussed by Bruns and Boyd in their editorial [3] that accompanied the publication of the evaluation study by Lenters-Westra and Slingerland [2] which was discussed in
• Episode 4 discessed the study by Lenters-Westra and Slingerland.
• Episode 5, Dr. Craig Foreback provided a discussion of measurement principles to help us understand both the NGSP certification protocol and the use of 3 different secondary reference methods in the Lenters-Westra and Slingerland study.
• Episode 6 provided a more detailed discussion of the statistical web analysis of the replication and comparison of methods experimental results.
• Episode 7 made use of error grids to compare different quality goals and requirements that might be applied to determine the acceptability of a HbA1c method.
• Episode 8 discussed how to prepare a Method Decision Chart to help judge the acceptability of performance.
• Episode 9, described a "mixed units" Method Decision chart that could be used to summarize all the performance results on these POC devices and showed that only the best precision and lowest bias provided the quality necessary for reliable patient testing.
• Episode 10, reviewed the performance of other laboratory testing methods as available from the latest CAP PT survey

Background

Erna Lenters writes that she has been working as an analyst at the clinical chemistry department of the Isala klinieken in Zwolle the Netherlands for 21 years. The Isala Clinics is one of the largest hospitals in our country and we serve approximately 1.2 million inhabitants of the Netherlands. The Isala Clinics has been a worldwide recognized centre of expertise for Diabetes for many years. Our hospital together with another hospital in the Netherlands (Queen Elisabeth hospital in Winterswijk) forms the European Reference Laboratory for Glycohemoglobin (ERL). Dr. Kor Miedema was the former director of the lab and he was also the former chair of the IFCC working group for the standardization of HbA1c. He retired and Dr. Robbert Slingerland is his successor. Robbert Slingerland is a Clinical Chemist and Laboratory Director at the Isala Clinics. He is responsible for diabetes testing and point-of-care testing and is Chair of the European Reference Laboratory for glycohemoglobin and member of the IFCC working group mentioned above. He is also member of the committee for Point-of-Care Testing of the Dutch Society of Clinical Chemistry. Furthermore, he is involved with the ISO working group TC212 and the revision of ISO15197 for blood glucose meters. I have been involved in the HbA1c standardization since the moment Dr. Miedema was asked to become chair of the working group in 1995. I am very grateful to both Dr. Kor Miedema and Dr. Robbert Slingerland for giving me the opportunity to develop my own line of research next to my work as a lab technician.

Our lab makes the calibrators for the IFCC primary reference methods by mixing approximately 99,5% pure HbA0 and  approximately 98% pure HbA1c (the worldwide gold standard). We also collect the blood for the secondary reference material which is send annually to all labs in the IFCC network (12 labs worldwide), and all labs in the NGSP network, Sweden, Japan and the manufacturers collaborating with the IFCC network. The next blood donation campaign will be on March 17th and 18th. Each time, approximately 100 patients with diabetes and 15 colleagues of mine will donate 250 mL blood. 90% of the patients come every year on voluntary basis to donate their blood. From the donor blood the lab in Winterswijk produces 8 pools of blood with varying HbA1c values and prepares them for shipment all over the world. Furthermore, the NGSP certification sets for the manufacturers are made from the same donor blood. This is done yearly, and this was the basis for the establishment of the so called "master equations" between the IFCC and the designated comparison methods (NGSP, Sweden and Japan).

The ERL has the following HbA1c methods:

Our intention was and is to use different HbA1c methods based on different principles (affinity chromatography, ion-exchange HPLC and immunoassay). As a laboratory we try to keep the analytical performance of the methods used in Zwolle and Winterswijk at a very high level, since these methods are used for value assignment in EQAS schemes (CAP etc.), NGSP certification sets for manufacturers, and evaluation studies like the one published in Clinical Chemistry and the ADAG study (as published in Clin Chem Lab Med 2008;46(11):1617-1623). We have done numerous evaluations of different HbA1c methods but before 2008 it came never to a publication, because we didn’t realize the possible impact of our work in the rest of the world. Robbert encouraged me to start writing and at this moment I am working on my PhD thesis.

Comments on Real World Learning Episodes

First of all, we agree with your assessment that clinical requirements should take priority over requirements for proficiency testing or external quality assessment programs. When I discussed the results of this study with our diabetologist, Professor  Henk Bilo, we both realized that there is a gap between clinical chemistry where the results are produced and the healthcare professionals who interpret the results. We think that the biggest problem is that most of the healthcare professionals don't realize the difference in analytical performance between lab and point-of-care HbA1c methods. For instance, most of the nurse practitioners see an HbA1c value as a true value and do not realize the uncertainty in the results. Therefore, we designed a survey to evaluate the knowledge among care providers about the HbA1c assay and our intention is to send the survey to a large group of healthcare professionals involved in diabetes care. It's important that the limitations of current point-of-care instruments and laboratory methods be understood by healthcare professionals as these may have important clinical implications. Clinical chemists can play and in my opinion must play an important role by providing healthcare professionals with the information they need.

We also agree that it is a serious shortcoming in the CLIA regulations that waived tests are not subject to proficiency testing, as we pointed out in the podcast we prepared for the website of clinical chemistry. Fortunately, most of the manufacturers of HbA1c point-of-care devices also consider this a shortcoming and they encourage their users to join external quality schemes.

A sample at a level of 6.5% would have been a valuable addition to our study, however when I started the study the application of HbA1c as a diagnostic parameter wasn’t broadly accepted. I decided to use a sample at low HbA1c concentration (about 5.2%) because HbA1c is also used in our hospital in subjects admitted with a myocardial infarction, to discover unknown diabetes, to assess the degree of metabolic control in subjects known with diabetes, and to study the correlation between HbA1c and mortality and morbidity on longer term on subjects not diagnosed as having diabetes. Thus, imprecision in the low range of HbA1c is very important in our hospital. In retrospect, the value of 11.2% in the high range was too high. However, we regularly used this value to evaluate the performance of our laboratory methods, and the CV of the low and the high HbA1c samples was more or less the same. As a result I made the assumption that the CV would be constant in the range in between those values as well, and did not include a third sample. This also limited the amount of material the manufacturers had to supply for our study. Unfortunately, as it turned out,  the CV for some of the point-of-care devices was not constant over the entire range of HbA1c concentrations. The studies we are performing at this moment and in the near future will include more carefully selected samples.

The inclusion of samples in the 10 to 12% HbA1c range (mentioned in episode 6) might seem strange at this point in time, but it was consistent with the distribution specified for the NGSP certification protocol at that time (2008 and 2009).

We appreciate your request for more statistical data, but we decided to limit the amount of statistical data in the paper in order to keep it comprehensible for the average reader. And as you mentioned yourself, our paper already contains a disproportional amount of statistics compared to clinical chemistry standards.

In our opinion, table 2 provides a clear overview of whether or not the methods would pass the NGSP criteria with different secondary reference methods and with different lot numbers. We tried to put as much data as possible in the table but, as mentioned before, did not want to provide too many data. Also, we could have replaced the regression line for a correlation coefficient and a SD of the residuals but in our lab regression lines are more often used for quality assurance, and it feels incomplete not to present them.

We took the mean of the three reference methods in the graph because in our experience the mean reflects best the "true" value. The secondary reference methods used in this study were also used in the establishment of the master equation between NGSP and IFCC. Our personal opinion is, that it would be better to compare the NGSP certification results from the manufacturer with the mean of the secondary reference methods, instead of just one method chosen by the manufacturer. However, it is not up to us to decide on the practical applicability of such a proposal.

It is impressive that you put all the statistical data in one graph! It will save journals a lot of space and therefore money if evaluating data will be presented in a method decision chart in the future. At this moment it is not generally accepted to present data in this way, but it might be in the future. The method decision chart is in our opinion of added value. However, the study design should be perfect and the statistics behind the method decision chart should be fully understood otherwise wrong conclusions might be drawn from the chart. The DCA Vantage is a good example in this case (CV of 1.8% or 3.7%).

The NGSP and the IFCC working group have done a great job and it is their merit that the interlab CV has decreased from approximately 20% in1993 to approximately 4 to 5% in 2009. The applicability of HbA1c has changed, and therefore other specifications for quality are necessary. The analytical performance of some (new) methods is not good enough. This is recognized by some of these manufactures, and they are improving their methods. Even manufactures from some very good methods are improving their methods. We know from experience that some new versions of cation-exchange HPLC methods have a total CV of 0.3% in the EP-5 protocol at a low HbA1c value (+/- 5.0% HbA1c) and a high HPLC value (+/- 11.0% HbA1c) with no bias! These methods can easily meet six sigma! We hope that clinical chemists will choose for the best performing HbA1c methods, because we think that patients with diabetes deserve that.

We want to thank you for this discussion and we hope we can combine our knowledge in the future!

References

1. Cox J, Cowden E, Krauth G, Li J, Ledden D. A Comparison of the DCA Vantage Analyzer and Afinion AS100 Analyser Using HbA1c Reagent. (Abstract)  Clin Chem 2009;55(No. 6 Supplement):A92.
2. Lenters-Westra E, Slingerland RJ.  Six of Eight Hemoglobin A1c Point-of-Care Instruments Do Not Meet the General Accepted Analytical Performance Criteria. Clin Chem 2010;56:44-52.
3. Bruns DE, Boyd JC.  Few Point-of-Care Hemoglobin A1c Assay Methods Meet Clinical Needs. Clin Chem 2010;56:4-6.
4. Westgard JO. Basic Method Validation, 3rd ed. Madison WI:Westgard QC, 2008.
5. Boyd JC, Bruns DE. Quality specifications for glucose meters: Assessment by simulation modeling of errors in insulin dose.  Clin Chem 2001;47:209-214.
6. Holmes EW, Erashin C, Augustine CJ, Charnogursky GA, Grysbac M, Murrel JV, McKenna KM, Nabhan F, Kahn SE.  Analytical bias among certified methods for the measurement of Hemoglobin A1c.  Am J Clin Pathol 2008;129:540-547.
7. Bland JM, Alterman DD. Statistical methods for assessing agreement between two methods of clinical measurement.  Lancet 1986;i:307-301.
8. CLSI EP27-P.  How to Construct and Interpret an Error Grid for Diagnostic Assays. Clinical and Laboratory Standards Institute.  Wayne, PA. 2009.
9. Clarke WL, Cox D, Gonder-Fredrick LA, et al. Evaluating clinical accuracy of systems for self-monitoring of blood glucose.  Diabetes Care. 1987;10:622-628.
10. Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose.  Diabetes Care 2000;23:1143-1148.
11. Westgard JO.  Six Sigma Quality Design & Control: Desirable precision and requisite QC for laboratory measurement processes.  2nd ed. Madison WI:Westgard QC, 2006.