PATHOLOGY 520: Selection, Evaluation and Control of Analytical Methods

Sample exam questions, part I
Sample exam questions, part II
Sample exam questions, part III

Note: Access to actual MT520 course is password protected; contact webmaster for details.

James O. Westgard, Ph.D. Dept. of Pathology and Laboratory Medicine University of Wisconsin-Madison See Dr. Westgard's biography

This course instructor is James Westgard, PhD. The CLS Program Director is Sharon S. Ehrmeyer, PhD., and the Chairman of the Department of Pathology and Laboratory Medicine is Michael Hart, MD. Pictured here are Tricia Barry, BS MT(ASCP) and Elsa Quam BS MT(ASCP).

Introduction

Pathology 520 is a 3 credit course intended for senior students majoring in Clinical Laboratory Science (CLS). The course format is in transition from traditional lecture instruction with hardcopy materials to internet course materials with discussion periods. This is the first year that internet materials are available for the entire course - method validation, basic QC, and QC planning. The internet materials are being provided on the website of WesTgard" Quality Corporation. This website has provided the resources to develop and support these training materials, courtesy of Sten Westgard who prepares the materials for electronic publication and manages the website.

Purpose

The course is intended to provide the background and skills necessary to quantitatively manage the analytical quality of laboratory testing processes. It focuses on three areas: the initial evaluation of the performance of a method; basic statistical quality control practices in healthhare laboratories; the planning or selection of appropriate quality control procedures.

Objectives

With completion of this course, students should be able to:

• Understand the current regulatory and accreditation requirements that influence a laboratory's practices for managing the analytical quality of testing processes;
  • Recognize the implications of proficiency testing regulations for setting laboratory quality requirements;
  • Access current information and materials from the World Wide Web.
• Utilize quality requirements for evaluating the performance of a measurement procedure and selecting a control procedure;
  • Identify sources of recommendations for quality requirements;
  • Apply clinical and analytical quality-planning models to translate quality requirements into analytical specifications for precision, accuracy, and quality control;
• Assess the performance of a measurement procedure from method evaluation studies;
  • Understand method evaluation experiments;
  • Perform the statistical calculations necessary for summarizing data from method evaluation experiments;
  • Interpret statistical summaries of experimental data to judge the acceptability of a measurement procedure;
• Understand basic statistical quality control practices;
  • Chart quality control data by established techniques, such as Levey-Jennings and Westgard multirule QC procedures;
  • Interpret quality control data to judge the acceptability of a routine analytical run;
• Select appropriate QC procedures based on quality requirements and the performance characteristics of measurement and control procedures;
  • Understand QC planning tools, such as power function graphs, critical-error graphs, and OPSpecs charts;
  • Use OPSpecs charts for manual QC selection of QC procedures to satisfy analytical quality requirements;
  • Download QC information, demonstration software, and program tutorials from the World Wide Web;
  • Use a computer program to select QC procedures that will assure the quality required by analytical and clinical quality requirements.

Internet materials

• This course description with schedule of lessons
• Lesson outlines and materials
• Web calculator tools

Required computer programs

• MicroSoft Excel spreadsheet (or similar electronic spreadsheet)
• QC Validator from Westgard Quality Corporation, Ogunquit, ME. A free educational version is available via download from this website. To obtain a password to download the program, please contact Dr. Westgard [e-mail jo.westgard@hosp.wisc.edu, telephone 608-263-1498, fax 608-263-1568].

Useful reference materials

• Middleton MR. Data analysis using Excel 5.0. Duxbury Press, Wadsworth Publishing Company, 10 Davis Drive, Belmont CA 94002, 1995.
• Middleton MR. Data analysis using Excel - Updated for Windows 95. Duxbury Press, Wadsworth Publishing Company, 10 Davis Drive, Belmont CA 94002, 1997.
• Westgard JO. OPSpecs Manual - Expanded Edition: Operating Specifications for Precision, Accuracy, and Quality Control. Westgard Quality Corporation, Ogunquit, ME, 1996.

Examinations and grade

Description of lessons

Lesson 1. How do you manage quality? In Myths of Quality, Dr. Westgard challenges current thinking that analytical quality is better than needed for medical care. In Understanding Quality, Jerry Ehrmeyer discuss the relative nature of quality and the need for "standards" to compare and judge quality. These standards of quality are central to the quality management process described by Dr. Westgard in The Management of Quality: The need for standard processes and standards of quality.

Lesson 2. What's the approach for selecting and validating a new analytical method? What's the approach for selecting and validating a new analytical method? In MV - The inner, hidden, deeper, secret meaning, Dr. Westgard reveals the key for developing an objective and rational process for establishing new analytical methods. Dr. Sharon Ehrmeyer clarifies the regulatory and accreditation requirements for establishment of new methods in MV - The Regulations. A rational process begins with careful selection of the analytical method, as described in MV - Selecting a method to validate.

Lesson 3. How do you validate the performance of a laboratory method? Specific experiments must be performed to estimate the different types of analytical errors that might be present. In MV - The experimental plan, Dr. Westgard discusses the organization of these experiments to provide an efficient plan for determining the types and magnitudes of analytical errors. The first experiments that are usually performed are described in MV - The linearity or reportable range experiment and MV - The replication experiment.

Lesson 4. What experiments are used to study the accuracy of a method? This lesson focuses on three experiments that may be used to study the accuracy or systematic errors of a method. A comparison of methods experiment is commonly used to estimate the average systematic error, or bias, between the new or test method and a comparative method. An interference experiment may be used to estimate specific sources of constant systematic errors and a recovery experiment may be used to estimate proportional systematic errors.

Lesson 5. How are the data analyzed statistically? This lesson reviews the statistical calculations that are appropriate for the data collected in the different method validation experiments. Web-based tools are provided to illustrate the calculations and graphs that are useful for different sets of experimental data.

Lesson 6. How do you interpret the statistical results from method evaluation experiments? This lesson describes how to use statistical parameters to estimate the specific types of errors of interest in a method evaluation study. It focuses mainly on the comparison of methods experiment where the interpretation is most difficult.

Lesson 7. How do you judge the acceptability of the performance of a method? This lesson provides the key to making rational decisions about method performance, which is to compare the estimates of analytical errors with the amount of error that is allowable. If the observed errors are greater than the allowable error, method performance is not acceptable. Acceptable performance requires that the observed errors be smaller than the allowable error.

Lesson 8. What's the approach for monitoring daily performance of a method with statistical QC? This lesson introduces the basic principles of statistical Quality Control (QC). In QC - The Idea, Dr. Westgard describes the basic approach for monitoring the variability of a method and identifying those situations where test results exceeds the variation expected under normal operation. Dr. Sharon Ehrmeyer reviews the USA laboratory regulations and accreditation guidelines that influence current QC practices.

Lesson 9. How do you set up a statistical QC procedure?. In this lesson, Elsa Quam, BS, MT(ASCP) reviews the purpose of statistical QC and describes the characteristics of control materials in QC - The Materials. She points out the importance of the matrix, stability, vial-to-vial variability, assayed vs unassayed, analyte levels, and pretreatment steps in the selection of control materials. A summary table of clinical decision levels is provided courtesy of Dr. Bernard Statland. In QC - The Calculations, Dr. Westgard reviews the calculations necessary for establishing control limits, including cumulative or lot-to-date statistics and control limits.

Lesson 10. How do you plot control results on a Levey-Jennings chart? In this lesson, Patricia Barry BS, MT(ASCP) shows you how to construct a Levey-Jennings control chart, plot control values, and interpret those results. A cholesterol example is used to illustrate the calculation of control limits, plotting of control ponits, and interpretation of results. The answer sheet shows the relative numbers of out-of-control situations when you interpret the control results with 1_2s vs 1_3s rules, and also illustrates the interpretation of these results with a 1_3s/2_2s/R_4s multirule procedure. Dr. Westgard explains the reasons for the different numbers of rejections by the different control rules in the lesson QC - The Chances for Rejection.

Lesson 11. How do you interpret control data using the multirule procedure? In this lesson, Dr. Westgard describes the use of multiple decision criteria or multiple control rules to interpret control data. Control rules that are commonly used in multirule procedures are defined and illustrated graphically. In QC - The Multirule Interpretation, a common multirule procedure, often knows as "Westgard Rules", is used to illustrate how to interpret multiple rules with multiple control materials and multiple analytical runs.

Lesson 12. How do you solve control problems? In this lesson, Elsa Quam, BS, MT(ASCP) identifies both good and bad habits for dealing with out-of-control problems. Patricia Barry, BS, MT(ASCP) discusses the importance of documenting the "history" of a method in order to learn from past experiences. Finally, there is a review of the overall process of establishing a QC procedure.

Lesson 13. How do you build quality into a laboratory testing process? In Putting Quality into Quality Control, Dr. Westgard emphasizes the need for quality planning, with particular attention to selecting statistical QC procedures that are appropriate for detecting medically important errors. Four practical approaches for QC planning are identified in Starting a QC Planning Process. A detailed example of QC planning using analytical quality requirements is provided for a Cholesterol test that is subject to CLIA proficiency testing. Another example considers a Cholesterol test and a clinical quality requirement based on NCEP guidelines for interpretation of a screening test.

Lesson 14. How can you do QC planning? In Tools and Technology for QC Planning, Dr. Westgard describes how to make QC planning practical using graphical tools, such as power function graphs, critical-error graphs, and OPSpecs charts. The QC Validator computer program is introduced as an example of the kind of technology needed to provide these graphical tools. An educational version of the program (QC Validator 2E) can be downloaded to provide these tools for the duration of this course. A demo of the program can be downloaded from the website and viewed to illustrate QC Validator's use. Detailed tutorials are also available via download from the website. FAQs about QC Validator provides additional discussion of the use and application of this computer program.

Lesson 15. QC planning tools - What are power function graphs? The performance of statistical QC procedures can be characterized by determining the probabilities of rejecting analytical runs that have different sizes of errors, as described in this lesson on Power Function Graphs. An example of the application of power functions considers Higher N QC Procedures for Immunoassays. Additional discussion is provided by FAQs for Power Function and Critical- Error Graphs.

Lesson 16. QC planning tools - What are critical-error graphs? This lesson on Critical-Error Graphs demonstrates the more quantitative use of power function graphs based on calculating the size of medically important errors, then imposing these critical-errors on the power curves to determine the probabilities for rejection. The application of critical-error graphs is illustrated by QC Selection for a Multitest Chemistry Analyzer that performs 18 different tests. A detailed application of the QC Validator program is illustrated for albumin via downloading the Multi-test Chemistry example.

Lesson 17. QC planning tools - What are OPSpecs charts? Operating specifications describe the imprecision and inaccuracy that are allowable for a method and the QC that is necessary to assure the desired quality is achieved. The lesson on OPSpecs Charts explains the origin of this powerful QC planning tool. The cholesterol examples for analytical and clinical quality requirements should be reviewed to understand the application of this tool. FAQs about OPSpecs charts provides further discussion to clarify their use and interpretation.

Lesson 18. QC planning tools - What are quality planning models? The theoretical and conceptual basis of the QC planning process and the QC planning tools are based on the expected relationship between a quality requirement and the factors that contribute to the variability of a test result. In this lesson on Quality Planning Models, Dr. Westgard uses the concept of an error budget to describe the relationships between analytical total error criteria with analytical factors and clinical decision criteria with preanalytical and analytical factors. Participants should review the two cholesterol examples that represent applications of these analytical and clinical error budgets (Cholesterol with Analytical Quality Requirement, Cholesterol with Clinical Quality Requirement).

Lesson 19. What is a Total QC strategy? The appropriate balance between statistical and non-statistical QC techniques depends on the performance available from the statistical QC procedure. In this lesson on Total Quality Control (TQC) Strategies, Dr. Westgard describes a flow chart that provides guidance in developing TQC strategies for your own tests. A review and summary of the QC planning process is provided by Dr. Westgard's AACC Online article Strategies for Cost-Effective QC.

Lesson 20. Applications - Can you really perform QC planning quickly - in one minute? This lesson focuses on Normalized OPSpecs Charts and makes use of a web-based calculator for normalized operating points to provide a quick and easy way to apply the QC planning process. You need to print a series of normalized OPSpecs charts, then use the web calculator to determine the normalized operating point to be plotted manually on these charts. A series of examples are provided to give you enough practice to become proficient in using these tools - proficient meaning that you should be able to perform QC planning within one minute.

Lesson schedule for 1999 UW-Madison students