Overview of the Statistical Analysis Process

The AASHTO re:source Proficiency Sample Program (PSP) is based on testing two samples that are nearly identical or have identical test properties.   Each sample in the pair is treated as a separate sample and is not combined during testing. The data reported for the two results are analyzed individually for the Reproducibility (between-laboratory) and combined for the Repeatability (within-laboratory) information. After the data collection process is closed, the data is refined to determine the consensus value.

The process is as follows:

1. All “unpaired” and “null data” is removed.

The repeatability data requires paired data.  Any participant that submits data for only one of the two samples has the submitted data removed from the analysis process. 

2. Using percentile rank, the inner 75% of the data is used to define the first “Core Data” set and the median is determined.   

3. Upon defining the first Core Data set, any value that falls beyond ± 4.725 standard deviations from the median is considered to be “invalid” and removed from further analysis. 

4. The process is repeated a second time using the inner 75% and the determination of a new median.

For the second refinement, data that falls beyond ± 2.7 standard deviations from the median is considered to be “outlying” and removed from further analysis.  The new data set is the final set used in the analysis. 

5. Using the “Final Core Data” set, the mean and standard deviation is determined for the each of the samples in the pair.

6. Z-scores and ratings are assigned for each participant who submitted data. 

7. Repeatability values are calculated for each participant who submitted paired data as follows:

Each participant has a data point calculated for them. The value is measured as the deviation in the difference between the participant’s data compared to the difference between the mean values of the set.

8. All participants who submitted data receive z-scores and ratings even if they submitted unpaired data or a portion of their data was removed from the analysis process. 

9. Additional values are calculated and determined for the data such as the 1s%, d2s, and d2s% and are typically used as measures of indices of precision for the test methods.  These values are not used as part of the data analysis process or for assigning ratings.

1s% = the standard deviation divided by the mean expressed as a percentage 
d2s (difference limit) = the standard deviation multiplied by 2√2 (≈ 2.8)
d2s% = the difference limit divided by the mean expressed as a percentage 

For a full description of the Data Analysis Process please view NCHRP Report 9-26 Phase 3.