Quality Requirements and Standards

In December 2014 and January 2015, we asked laboratory professionals around the world to share their views on analytical goals. Here are the results.

Analytical Goal Survey Results

Sten Westgard, MS
January 2015

A link to this survey was posted on the home page of Westgard Web, the Westgard Rules blog. The survey was announced in the December issue of the Westgard Web newsletter, which is sent to over 20,000 laboratory professionals. Similar announcements were made via Twitter, LinkedIn, and on the [ML] listserve and another listserve in the United Kingdom.

More than 450 responses were received from more than 80 different countries:

Type of Institution

The majority of survey respondents were hospital laboratories. Most respondents were from "working labs" and not vendors or educational institutions.

Size of institution

Most of the respondents were from what we consider to be "large" hospitals: more than 500 beds and/or more than 100,000 tests in annual volume. Some might see the pairing of the number of beds and test volume and think these are not correctly synchronized. However, test volumes in the West are much higher than those volumes elsewhere in the world. Some hospitals in Asia may have many more beds but a smaller volume than we would expect from a US hospital. The main aim of this question is to sort out the big from the small labs - and we can see that a majority of responses are coming from the large laboratories.

 The Use of Analytical Goals

Most labs use analytical goals to evaluate method performance, monitor their PT/EQA compliance, compare their performance with peer groups, and design their QC. A minority of labs use these goals to help them select their methods, measure their uncertainty, or assess their quality on the Six Sigma scale. A very small number of respondents indicated that analytical goals were not used for any of these purposes. We can assume that that number may be artificially low. From my anecdotal travels around the world, I see a far higher percentage of laboratories that do not use any goals in their operations.

The Type of Analytical Goals Used

The vast majority of laboratories use Allowable Total Errors, as well as goals for SD based on biologic variation. Surprisingly, the manufacturer claims for performance. Target measurement uncertainty goals are only used in about 15% of labs, less still for Root Mean Standard Deviation (RMSD). Only a very few labs use goals defined through a clinical outcome model.

The Sources of Analytical Goals

A majority of labs derive their goals from the PT and EQA surveys and peer groups, use manufacturer claims, and use the goals derived from the biologic variation database ("Ricos Goals"). Just under a majority of laboratories use CLIA goals, showing the wide influence of US regulations reaching. Fewer laboratories make use of their own calculations, their historic/traditional goals, or determine goals through surveys of their own clinicians. The RCPA and German Rilibak rules are used in around 15% of laboratories.

The Changes in Goals Labs would like to see

The biggest item on the laboratory "wish list" is to develop new goals based on the intended clinical use of the test. This is the type of goal that is currently least-used, as earlier questions noted, but laboratories recognize that this is the most useful and relevant type of goal to implement. Next on the list is the laboratory desire to have a common, harmonized set of goals for PT and EQA. After that, labs would like more expert groups, clinician surveys, and an update to the CLIA goals. Interestingly, there is a stronger desire to widen goals than to tighten them, reflecting the struggle between finding goals that are practical while setting goals that will spur improvement in the test quality. There is less interest in adopting alternative goal-setting approaches, such as target measurement uncertainty. And the least desired change is to eliminate the use of Total Error. Since most laboratories use this type of goal, indeed nearly all PT and EQA surveys express their goals in this format, it is not surprising they are very reluctant to abandon this type of goal.

Desired Support in the Use of Analytical Goals

 Most laboratories overwhelmingly want help in applying analytical goals, but a majority of labs want help in every phase, from the definition of goals, to the databases and sources where goals can be found, to a simpler way to select goals. And most labs want to expand the use of goals into the pre-analytical and post-analytical phases of the Total Testing Cycle. The most relevant conclusion from this question is that labs are more interested in applying goals than in arguing about the definition of the goals. They have a bias for action and implementation, not debate. The literature and many conferences seem to have these priorities reversed; they prefer to debate and invent more complex models, while ignoring the realities of how these goals must be implemented. It is, to be sure, much more academically interesting and rewarding to tackle the subject of defining a goal, rather than the reality of having to implement it.

For each question, the survey also had a place for comments or "other", and those additional entries are discussed in another article on the website.

A note on the validity of this survey

Given the way this survey was conducted, there can be no doubt that it suffers from numerous biases. It suffers from a geographic selection bias due in part to the Westgard Web newsletter (>20,000)  and the LinkedIn connections (>2,000), which comprised the majority of the distribution channels for the survey. The Westgard Web Newsletter subscribers and connections  subscribers are not equally distributed across the globe; both have a large number of US members. Furthermore, we can assume that those who decided to take part in the survey are part of a self-selection bias, in that they must already be interested and more engaged in quality goals (than the normal lab population). So the group may be less representative of the "true" laboratory world (as we noted earlier, very few of the respondents indicated that no goals were in use.). Another bias that affects each question is a framing bias: the way each question is constructed impacts the answers that respondents give. Again, this survey probably understates the number of labs that don't use any goals or don't want any changes to goals, etc. - there is an implicit pressure to choose some answer even when no answer may more accurately reflect the respondent's lab.

These bias issues are not unique to this particular survey, but are typical to all such surveys. Nevertheless, we believe that thes results provide a decent snapshot of global laboratory opinion on analytical goals.

We hope you find this useful in your work. Also, for those who wish, a chart-pack can be downloaded for free with graphics of all the survey results.