Tools, Technologies and Training for Healthcare Laboratories
  2. Home
  3. Essays
  4. ISO
  5. MU Survey 2015: Labs Speak Out About Uncertainty

ISO

MU Survey 2015: Labs Speak Out About Uncertainty

Here are more than 120 personal comments from the participants in our Global Measurement Uncertainty Survey. Some labs have no ambiguity about how they feel uncertainty. Here are the responses from all the countries outside the US (more than 85 countries from every civilized continent and region in the world). Believers, Beleaguered, Compliant, and Exasperated - just some of the categories of replies.

Labs Speaking Freely about MU: Open Comments from the 2015 Global Measurement Uncertainty Survey

Edited by Sten Westgard MS
November 2015

In the Fall of 2015, we opened up a survey to gather the views of laboratories around the globe on measurement uncertainty. As is always true for Westgard Surveys, we end the survey by giving every participant an opportunity to speak freely, to add additional comments, questions, compliants and more. Often these responses reveal far more about the state of the laboatory than the simple survey questions do.

Here, then, are the comments from more than 120 respondents of the survey. These comments have been edited for clarity (I have corrected spellings and grammar where I thought I could discern the intent of the writer). I have also organized them into arbitrary categories, trying to group them in an logical fashion rather than simply present them in random order. I apologize if these categories are harsh. Note that these are all comments from outside the US; the US comments on measurement uncertainty are very different and will be posted on a different page.

The Believers

  • Calculating uncertainty measurement is very useful for laboratory to assure the accuracy of laboratory report.
  • In my opinion, uncertainty is useful in clinical laboratories, as it is in calibration laboratories. However, education is necessary.
  • Clinicians are not well educated about MU, the challenge is their education!
  • I hope that the uncertainty of measurement will be used above all in the daily practice of quality control internal and external, that will change the mode to decide if the data obtained are "in control" or "out of control", I think we need to overcome the rules control that are based on the theory of Gauss, in fact the distributions of the control data often do not follow this type of distribution, because who manages the control does not understand that the average or standard deva[tion] has changed and it is necessary to intervene on the parameters of media and GAUSS standard deviation
  • Estimating uncertainty allows the laboratory to know its accuracy, otherwise only their variation and bias is known. Likewise, the healthcare worker best known measuring process
  • We don't have MU on our reports but the number of significant figures we use are based on MU (see Badrick & Hawkins) Clinicians (and patients) are often concerned about small test changes until they are informed that they are within the confidence limits of measurement uncertainty.
  • It is a very important estimate, but should begin deploying use from the academy and its inclusion in the INTERPRETATION of the critical values with the medical team.From the metrological point of view is the correct estimate of the accuracy.
  • I think is very important use the measurement uncertainty
  • The measurement uncertainty is helpful for the decision of clinicians especially for the analytes which have the very low detection limits. I have seen when there were even the small differences from reference limits, the clinician can decide that there is pathology which causes serious concern to the patient. I am a technical expert for ISO 15189, and going to the labs for the ISO 15189 assessment. I think that the requirement for estimation of MU of a measurement result, especially at the critical level by using the IQC and EQA values can give confidence to the laboratory for its performance. However, it is not easy to estimate the MU, but it is valuable if it is calculated for long term. I realized that the kit manufacturers were not happy for supplying the calibrators’ uncertainities.
  • As I have worked at a NMI for many years, it is a fundamental part of my work and I have grown to appreciate the usefulness of MU for identifying the sources of imprecision and bias. It also provides an indication of if you are in control, and can have confidence in the method you have developed. I can recommend the following publication: http://www.lgcgroup.com/our-science/national-measurement-institute/publications-and-resources/technical-reports/evaluating-measurement-uncertainty-in-clinical-che/#.VhQS70pwbRN
  • [I] am responsible for performing all analysis in a specialist haemostasis laboratory in one of the largest hospitals in the UK. [I] recently gave a talk about uncertainty of measurement at a national conference and was surprised about the lack of knowledge out there in other hospitals. I would be very keen in continuing my involvement in any further work that is produced here.
  • [U]ncertainty is important
  • [I]t was helpful for us to explain the differences in results between 2 analyzers (Architect and COBAS 6000)
  • I believe MU is a vital piece for the implementation of [I]QCP (Risk mitigation process)
  • MU is also useful as part of the evaluation and validation of a method and/or analytical equipment.
  • ....I think that MU is a laboratory issue, and its normal the clinicians [don’t ask] for it, for they dont know [how] to [apply] it to rcv, or clinical cut off decisions . That information must be delivered by the laboratory. Using maximum allowable MU, allows the laboratory to implement icqp that detects the problem on the moment of the control (instead OF detecting the problem when evaluating the Total Error (weekly, monthly etc)....
  • [G]ood to know to guide clinicians when asking about the significance of 2 follow-up results.

The Beleaguered

  • It is difficult in the economic and political situation of the country, [to] calculate uncertainty [of] measurement, when we have problems in the structure of the public health system because it is not invested in quality control, and quality management is not a priority.
  • Have never had a Dr specifically request MU data for any test.
  • Don’t see why we cannot use the QC terminology instead of the metrological terminology which just confuses everyone. There is a draft ISO standard ISO/TC212 committee headed by Graham White.
  • Clinicians seem to be unconcerned about uncertainty of measurements and that reduces its effectiveness.
  • Physician[s] [do] not know the concept of MU
  • Not sure why this is being imposed on all labs - it's not really representative of how we work in a hospital lab. Is our normal method of monitoring CV not sufficient?
  • MU is not in the field because it confuse both patient and doctors. In the, lab ISO 15189 didn't give enough explanation of how to calculate/use/limitation or interpretation.

The Cautious

  • I think MU is still a very theoretical concept yet and even those that are updated in ISO 15189 don't understand it well. It has been a confusing concept until the last versión of ISO 15189 [that] is explained in terms of %CV but it will need to be explained better to apply it properly. People need more basic QC because all relies on knowing the basics to apply more complex concepts.
  • I believe that if MU is published in reports it would cause confusion for the clinician.
  • Recently I took part in the meeting diagnosticians from the region where one of the speakers would encourage us to use the MU. None of about 50 people in the room intend[s] to in the near future to calculate the MU because they fear that doctors and patients do not understand the MU and may accuse us that our tests are inaccurate and imprecise :-)
  • I think t[h]at the lack of use of uncertainty [is due] to the [challenge of] medical acceptance of the concept .... They would not know how to manage with the information added to the results.

The Challenged

  • [Measurement uncertainty] is useful when interpreting results: however, the current laboratory information system can not handle the reporting of MU; it may also help the laboratory when acquiring / choosing instruments for the Lab; at this time, it needs standardization which is a big task.
  • [T]his concept has yet to reach veterinary medicine in any meaningful way. Most labs not interested, ignorant of its use or intent, most clinicians the same. Many barely understand any uncertainty or probability associated with a test result.
  • [T]he proper place for use of MU (in labs measuring biological quantities) is in SOPs (& training) to ensure lab personnel can offer advice to clinicians plus in LIMS (automatic rules can be programmed to prompt for repeat tests/samples (& the same can be done for Reference Change Values). Most LIMS are not configured to add x +/- MU and most/all clinicians wouldn't know what to do with this anyway
  • We would be happy to use MU but the terminology and complexity of integrating this into our routine lab processes makes this quite daunting
  • Our main problem is mostly about choosing the appropriate goals.
  • Main issue found with bottom up is how to decide what is an acceptable uncertainty.
  • Difficult concept for clinicians. For Troponin it is understood.
  • It will take practice to understand its utility

The Compliant

  • Majority labs do within and between runs CV as minimum for MU.
  • [W]e do it annually for regulatory requirements then forget it until next year ...
  • Can you explain benefit from measurement uncertainty in laboratory[?] Because in our laboratory, measurement uncertainty just tools to comply ISO 15189. Clinician[s don’t] understand about uncertainty, if we report in result patient, that [may] cause [m]any question from clinician. An[d] how implementation uncertainty in USA[?]
  • Although the ISO17025/15189 requires MU, it is not requested during the inspections in Germany.
  • We estimate the MU for long list of analytes solely to meet the ISO 15189 requirement. We would never have done the calculation if it was not made compulsory. Am not so sure about other settings but in my hospital, clinicians seems do not bother about it. That is bad, but even worse when we sometimes get confused and uncertain about the estimation that had been documented.(Is it requires some kind of awareness?) How [I] wish there were some simpler approach to substitute current one to estimate the MU.
  • [M]andatory to calculate MU for accreditation in France but not used in daily practice
  • It's done only because it's a lab accreditation requirement. No physician has ever asked for this and may not even know what is it. They have their own way of interpretation get a numerical value.
  • My previous lab was ISO 15189 accredited. We spent close to 200 man-hours completing these calculations and I personally spent 40 hours creating an excel template to automate calculation and another 40 hours training the lab staff to use the excel template. Our lab was accredited, and the calculations were never looked at again and continue to sit in a tidy fashion in a binder in the lab. The value of the numbers depends on the mode of calculation. There is a lack of standardization for what is included and not included in these estimates. Some overestimate, some underestimate for that reason. Until we can all agree what to include and what to exclude in the equations to calculate MU, I don't think any lab will actually see utility in this exercise.
  • 1. NABL India has included UM in the recent guidelines. We were told that that this would become mandatory over the next two years.
    2. Most of the laboratorians and assessors that I have come across opine that calculating uncertainty of control measurements is sufficient.
    3. Fascinated as I am with numbers and not happy with just 1.98*CV, I had included uncertainty of sample measurements (encompassing the time period of storage as per our departmental protocol) and calibrator uncertainty too.
    4. Nonetheless, I am still confused and unsure if I should do this exercise; but our NABL seems to insist on this and I have no choice.
  • We calculate MU because we're required to for ISO but we don't use it in any meaningful way.
  • We a[r]e keeping up to date mu, it[‘]s just that we are keeping it on file as per ISO req. but as of my entire career here in UAE no one ever called and ask about this.
  • I am Quality manager for our pathology dept. The technical service managers for each Pathology department are responsible for their teams UoM measurements
  • 15189 does not require to include the U in the test report. It is going to take time for this to be ...r[o]utine

The Confused

  • Which should be used as the acceptance criteria for monitoring MU: ALP from RCPA, ALE from CLSI or TEa from Dr. Ricos? [Of course, mu needs its own set of target measurement uncertainties. It’s not metrologically proper to compare calculated mu against essentially total allowable error goals]
  • The information available about MU is confusing and not easy to understand if you are not a mathematician. I have tried to understand exactly how to calculate it but I'm still not clear about it. Based on advice from our consultant, we plan to use IQC to calculate MU.
  • MU is compared to sigma value. [Given that mu will vary greatly and Sigma-metrics generally run from only 0 to 6, these are not comparable terms.]
  • We don't know ho[w] to calculate the uncertainty and why is it ne[ce]ssary?
  • No idea how it[‘]s calculated, interpretation no[r] its significance.
  • [S]till not clear procedure to calculate MU.
  • I am unsure with regards to calculating uncertainty of measurement
  • [If] MU is truly need[ed] for the clinicians, [do] all clinicians know what is MU?
  • I find the explanation and calculations for MU in the CLSI document very confusing and fail to see how it can be more useful than monitoring, providing the CVs.
  • This is an extremely confusing practice as there are several ways of calculating. Since we often look for subtle differences between groups containing large numbers of individuals, we like to have uncertainty of measurement estimates across the measuring range of the assay system.
  • The requirement and use of MU is unclear. Apart from the clause in ISO15189 requiring the calculation of MU the use of it is not mentioned or required in another area within the standard. Clarification on it uses would be helpful•

The Exasperated

  • If it was not for accreditation agency requirements UM would not even be on the radar screen.
  • This is malpractice and totally non- professional. Any professional, [who] enjoy[s] working in the laboratory will never prefer measurement uncertainty.
  • Theoretically I believe use of measurement uncertainty is important, however it's not applicable for daily routine test such as basic biochem[istry], CBC. I don't see the importance.
  • I understand the concept and intent of MU, but have concerns about implementation. Methods of calculation do not apply universally, particu[larly] to coagulation ,which is my specialty. Also have concerns about it being used as a defacto measure of quality by NATA assesors.eg comment from one assessor, we had to improve our MU for a particular assay, RcoF, which is accepted as being a dreadful test with CV on QAP surveys of approximately 30 % . Our QC was well within the manufacturers requirements.
  • Since ISO 15189 was 'spawned' out of ISO 17025 uncertainty of measurement is one aspect of the regulations that is "overkill" since it ignores the fact that that blood as a material is not inert.
  • The next point involves analysers and equipment - manufacturers are producing technology to a very high specification for FDA approval prior to it being available for the market place so why do we as end-users have to verify what has already been done (another hang up from ISO 17025 ?).
  • Complete waste of time, it seems to serve no real purpose, just something else to waste our time doing for ISO. There appears to be a number of different methods for working it out, no one single sensible way. All departments seem to do it in a different way, all companies you ask all have different methods. Yet another thing to do required by ISO but not consistently requested by all ISO examiners. Really necessary? Does it actually help the patient - probably not!
  • Actually, we think that the measurement of uncertainty is not valued for our clinicians, as a tool in the assessment of your patients.
  • We have not found any requirements to achieve regarding uncertainty, so [it] does not make any sense to keep it measuring it.
  • The majority of clinicians are ill informed as to measurement uncertainty and until they are fully educated, which I estimate will take approx.[imately] 10 years, then they can start to use it and we can then embar[k] o[n] fully evaluating MU - so until then it will remain a totally useless bureaucratic obligation.
  • Major waste of time for us. No one looks at beyond lab.
  • GUM is interesting but is not appli[c]able to lab results

The Hopeful

  • No laboratory can function efficiently without quality control and assurance. Even in these parts of the world we try our best in order to provide proper and correct diagnosis.
  • [I have] interest in mu
  • I feel that Clinicians should be given more information about MU
  • We are looking forward to begin measurement uncertainty

The Thoughtful Responders

  • Measurement uncertainty mainly depends on measure[e]ment precision (sometimes BIAS), because every other component (like MU of calibrators, pipettes etc.) is very small and is not contributing to expanded MU much, so in my experi[e]nce, it is not essential for method evaluation and monitoring.
  • ISO15189 requires knowing the measurement uncertainty, but does not give a guideline how to establish it, what factors should be included. This results in very different values for MU.
  • MU is an important concept, but needs to be less complicated so that it becomes useful in a routine lab
  • Bias [is] not incorporated in the MU estimation at the moment
  • Laboratories need guidance on how to work out the MU in all areas of pathology including microbiology and cytology, what it means practically and clinically, what it means in molecular such as in pcr, sequencing and NGS and how to determine them. A standardised approach would be ideal. Also, guidance on educating the service users is absolutely essential. Lastly, suggestion on funding/resources to get this work done.
  • My opinion, the physicians don’t need the MU information because they don’t have anything to do with [it]. The fact is that the analytical uncertainty as well as other sources of uncertainty including biological and pre-analytical uncertainty are already reflected in the clinical guidelines as “clinical uncertainty”. The terms such as predictive values of positive and negative results, ROC curves, gray zones, and borderline areas are in part the effect of analytical MU. If there are less analytical MUs, then the predictive values will be higher, and the gray zones/borderline areas will be tighter; i.e. the clinical uncertainty will be lower. The analytical improvements in Hb A1C assays is a typical example. The reduced analytical MU and consequent improved reliability of the results made expert organizations to revise the guidelines and recommend Hb A1C for diagnosis. Such an improved clinical uncertainty is seen with hs-troponin assays. Therefore, since analytical MUs are already addressed in clinical guidelines via determining decision intervals/cutoffs, reporting MUs to physicians not only is excessive, but could be harmful. It is “excessive” because there is no clinical guideline which asks the physician to take decisions based on analytical MU (Does anybody know any clinical guideline which needs MU for decision making?). And it could be “harmful” because it may make confusion when making clinical decisions; in that it is probable that a patient’s result is above the decision limit and treatment is indicated for him based on the guidelines, but his physician becomes reluctant in starting treatment just because the ‘result minus MU’ falls in the lower side of the action limit; and in contrast, it is probable that a patient’s result is below the decision limit and treatment is not indicated for him, but his physician hurries in treatment just because the ‘result plus MU’ falls in the upper side of the action limit!
  • I think sometimes it is rather difficult to approximate what is a good enough MU for a test. Personally I compare the bias and the imprecision with biological variation /bias. Because we have rather few contacts with clinicians it is not easy to know about the patients as to the result or difference between serial results is ok. How should DELTA CHECK criteria be set up? [B]ased on biological variation?
  • We consider modular approaches are useless in medical laboratory. All the approaches we are using are empirical, and are also applied to nominal testing.
  • I think MU is not appropriate for medical testing where a high range of biological variation exists.

The Trying-to-be-Helpful

  • A physician called to find out if he needs to do something for his patient as the urea result was 8.5 mmol/L in 2 consecutive visit. If I had calculated MU at that time (which was in year 1992) I could have helped him. Now I have calculated MU and nobody ask[s] for it. The physician was from India.
  • I frequently tell our clinicians that there is uncertainty in the measurements that our laboratory reports. As a Bone Marrow Transplant Laboratory performing CD34 analysis on haemopoietic progenitor cells, it is very difficult to get the clinicians to understand the large uncertainty in the no. of CD34+ cells per kg and quantification of donor / recipient chimerism post transplant.

The Begging-for-Help-ers

  • Assessing fitness for purpose is required by NATA/RCPA - no guidance on how to do this or what goals to use
  • [W]e are still unclear about correct calculation of MU!!!
  • The importance [of mu is] not yet understood.
  • [W]e want to do but we don't know how to do this
  • It[‘]s a new topic to be considered by our laboratories. We have to do this for ISO 15189 accreditation. There are no standards for "acceptable" measurement uncertainty in our discipline
  • There [are] no specific guidelines for our lab to use since we'd applied on drug test which is semi-quantitative test. Please guide us to improve our performance. Thanks.
  • This laboratory conducts a population screening test which includes sourcing information from external sources. Therefore, MU can only be applied to one part of the testing, ie laboratory analysis performed inhouse. How can MU be applied in such situations?
  • I would like to know how can I access the documents and guidelines about measurement uncertainty. I like to know if I made the correct choice [of] measurement uncertainty.
  • Facing challenges in how to calculate mu for HIV viral load test
  • Needs Practical training to understand QC in Laboratory applications daily practice
  • It is difficult to calculate measurement uncertainty for the unmeasured preanalytical factors
  • I find it difficult to reconcile U of M and indeed the Westgard rules with real time PCR. It is time for somebody to address PCR.
  • Worked examples are the best way to show how to calculate, report, and apply MU. Very complex mathematical models in GUM do not cover clinical examples.
  • An incredible amount of ignorance out there as to what MU is and why.
  • Seek more knowledge and articles on this topic
  • How it is going to affect the diagnosis if we do not apply MU[?] [T]here is no proper guideline how much MU is allowable for any parameter
  • How to include bias in MU?
  • [Woul]d like to get information about MU in [q]ualitative testing (elisa, elfa, pcr, se[q]uencing... [etc.])
  • I would like to know [a] simple way of calculating combined measurement of uncertainty.
  • It [would be] very helpful if there is a standard procedure for uncertainty measurement
  • Required training not available

The Just-Getting-Started

  • We aim to calculate MU annually
  • In this moment the laboratory is implemented the measurement uncertainty. For this we are introduced the concept of internal control, for the first step.
  • Have only recently had to calculate this data (within the last month) to pass UKAs accreditation
  • In my laboratory, I will implement it without any commitments to Government or any regional institution as an test project
  • In process to apply MU.
  • We have only just begun our MU journey so haven't really got to grips with it. There is a lot of discussion amongst quality managers as to which sort of calculations to use
  • Currently we have only determined UM for quantitative assays. Semi-quant and qualitative are pending.
  • We are at the early stages of implementation, less than 1 year in
  • We have been incorporating MU for the last two years.

The Doing-Something-Else-ers

  • [W]e usually do daily internal quality control for biochemistry investigations using commercially prepared normal and pathological controls. Westgard rules are checked and if outside the rules calibrate the reagents and re run the normal and pathological quality controls. [W]e are also participating external quality control program with the reference laboratory of Medical Research Institute, they send specimens every 3 months.
  • We really only keep under surveillance impre[ci]sion and inaccuracy on a monthly basis. This is our best approach to uncertainty.
  • We are not very interest[ed] in our measure[e]ment uncertainty. But when we validate our methods we determine the capability of the methods and we follow it each month. The bias is determine with patients comparisons (passing bablok) when we validate the method and the bias is the difference between our mean and the mean of a peer group with the CQI for the monthly following.
  •  ....What we use in the lab are the results reported by the SKML in the Netherlands, in fact the VC% of the analytical procedures obtained by the IQC measurements over a long period. This is according to the ISO15189.

Thanks to all who provided comments. It is invaluable to see the spectrum of knowledge and experience of measurement uncertainty in laboratories. Often we hear the idealized vision of what measurement uncertainty promises to achieve for laboratory medicine. These real-world responders provide a sobering reality check. If measurement uncertainty is going to have a true benefit to laboratory medicine, there's a lot of education and work that needs to be done.