|The Data Analysis Tool Kit|
|Written by James O. Westgard, Ph.D.|
Are you less frightened of statistics when we talk about them as tools? How about talking about statistics without showing you any equations? Well, that's what this lesson by Dr. Westgard does. If you can think about method validation as a job that needs a set of tools, you're ready to read this article.
This lesson is actually about statistics, but I didn't dare put "statistics" in the title. Many people get uncomfortable at the mention of "statistics". Others become uncomfortable when they see the equations for the statistical calculations. By now - three sentences into this lesson - you may be wondering if you can just skip the lesson and avoid the topic. The answer is NO; you need statistics to make sense of the data collected in method validation experiments.
To reduce the mental roadblocks in understanding statistics, there aren't any equations in this lesson. Instead, we're going to assume the calculations can be easily performed with the calculator and computer technology that's available today. Your main job will be to recognize what calculations are useful for different sets of data.
When I lecture on this topic, I begin by showing the class a bunch of tools, such as a hammer, wrench, saw, and screwdriver. Office tools (such as a stapler, scissors, paper, and pen) would provide just as good examples, but you're too comfortable with those tools. I want you to learn that you can use tools, even if you're not comfortable with them. So, let's consider the hammer, wrench, saw, and screwdriver.
You don't have to be an engineer, mechanic, or carpenter to recognize which tool fits these jobs. Everyone makes use of these tools to do certain basic jobs. While there are more complicated applications that take more skill and knowledge - and sometimes more specialized tools, everyone is capable of making practical use of the common tools.
It can be the same with statistics!
Statistics are just tools for combining many experimental results and summarizing all the data in just a few numbers. Remember that the objective of each experiment is to estimate the amount of error from the data collected. The key with statistics is to know which ones will provide useful information about the errors of interest in the different experiments.
Before trying to estimate these errors, we need to define the usable analytical range (or reportable range) of the method so that the experiments can be properly planned and valid data can be collected. The reportable range is usually defined as the range where the analytical response of the method is linear with respect to the concentration of the analyte being measured.
Then we start with the error analysis. First, we want to know the imprecision or random error from the 20 or more data points collected in a replication experiment. Then we need to estimate the systematic error from the 40 or more data points collected in a comparison of methods experiment. Finally, we need to make a judgment on the performance of the method on the basis of the errors that have been obesrved. The statistics are used to make reliable estimates of the errors from the data that have been collected.
Here's a picture of the tool kit you need to analyze the data from basic method validation experiments. The tool kit includes several calculators and plotters:
Note also that these tools often include both calculations and graphical displays of the data. There is an association between certain calculator and graphs because they complement each other for describing and displaying a set of data. For example, distribution statistics are use together with a histogram plot to descibe and display data for imprecision or random error. For inaccuracy or systematic error, regression statistics are used along with a comparison plot, or t-test statistics are used together with a difference plot.
Note also that there s a natural order for using the tools, as suggested by their location in the tool kit. Those at the top are generally pulled out first, e.g. the linear-data plotter is used in the beginning to establish the reportable range of the method, after wich the SD calculator will be used to estimate the imprecision or random error, whose acceptability can be assessed using the decision calculator. After these steps, the paired-data calculator will be used to estimate the inaccuracy of the method and the decision calculator used again to assess the overall performance of the method.
These calculator tools may be obtained from hand held calculators (e.g., Texas Instruments), electronic spreadsheets (e.g., Excel, Lotus 123), common statistics packages (Minitab, SAS, SPSS), specialized method validation software written for laboratory applications, and also from interactive web-calculators on this website. Many of these sources will also provide appropriate graphical displays, or you can construct them manually using graph paper. The Method Decision Chart should be constructed manually using graph paper.
We will provide more detailed discussions of the statistical calculations in other lessons, as well as the fine points of what the statistics mean and how they should be interpreted. For now we're going to focus on the bigger picture - which tools are appropriate for the different method validation experiments.
Given a set of experimental data, you need to recognize which tool is right for that job. Here are some general guidelines:
Internet calculators for educational use are available. These web-tools should be useful for working with example data sets and problem sets. However, they are not intended to answer all your data analysis needs for method validation studies. It is also recommended that you acquire your own calculator tools, either a general statistics program, a specialized method validation program, or an electronic spreadsheet.
The linear-data plotter is used with the data collected in the linearity experiment, where the purpose is to assess the analytical range over which patient results may be reported. The response of the method is plotted on the y-axis versus the relative concentration or assigned values of the samples or specimens on the x-axis. The “reportable range” is generally estimated as the linear working range of the analytical method.
The SD calculator is used for the data collected in the replication experiment, where the objective is to estimate the random error or imprecision of the method on the basis of repeated measurements on the same sample material. The statistics that should be calculated are the mean, SD, and CV. Also be sure to record the number of measurements used in the calculations.
The paired data calculator may be used with the pairs of results on each specimen analyzed by the test and comparison methods in the comparison of methods experiment. This is the most complicated part of the statistical analysis and requires the most care and attention. Linear regression statistics may be used along with a comparison plot, or t-test statistics may be used along with a difference plot.
The regression statistics that should be calculated are the slope (b) and y-intercept of the line (a), the standard deviation of the points about that line (sy/x), and the correlation coefficient (r, the Pearson product moment correlation coefficient). You may also see the slope designated as m, the y-intercept as b, and the standard deviation as sresiduals, respectively. The correlation coefficient is included to help you decide whether the linear regression statistics or the t-test statistics will provide the most reliable estimates of systematic error.
The t-test statistics of interest are the bias, SD of the differences, and lastly, something called a t-value which also requires knowledge of the number of paired sample measurements. Again, be sure to keep track of the number of measurements, which for the comparison of methods experiment is the number of patient specimens compared.
The decision calculator is used to display the estimates of random and systematic errors and judge the performance of the method . Therefore, this chart depends on the estimates of errors that are obtained from other statistical calculations. In brief, the chart is drawn on the basis of the quality requirement that is defined for the method and shows the allowable inaccuracy on the y-axis versus the allowable imprecision on the x-axis. The observed imprecision and inaccuracy of the method are then plotted to display the method’s “operating point” (y-coordinate is the estimate of inaccuracy or SE, x-coordinate is the estimate of imprecision or RE). The position of this operating point is interpreted relative to the lines that define areas of “poor,” “marginal,” “good,” and “excellent” performance. See the PDF files for details.
A note about our online tools
Our online calculators are set up for a fixed number of data points, e.g., 20 points for the SD calculator and 40 points for the Paired Difference, Linear Regression, and Correlation calculators. These calculator tools should be useful for example data sets and problem sets included with these instructional materials. However, they are not intended to answer all your data analysis needs for routine method evaluation studies. It is also recommended that you set up your own calculator tools using an electronic spreadsheet.