Road Safety Audits (RSA)

< Previous

Table of Content

Next >

Appendix A: Detailed Overview Of Evaluation Methodologies

Empirical Bayes Methodology

The empirical Bayes (EB) method can be used to account for regression-to-the-mean bias in before-after studies. Other advantages are that it can be used to accomplish the following:

• Overcome the difficulties of using crash rates to normalize for differences in traffic volumes between the before and after periods.
• Reduce the level of uncertainty in the estimates of safety effect.
• Provide a foundation for developing guidelines for estimating the likely safety consequences of contemplated installations.
• Properly account for differences in crash experience and reporting practices.

The premise of the EB method is to estimate the expected number of crashes that would have occurred in the after period had the treatment not been implemented (π) and compare that with the number of reported crashes in the after period once the treatment is actually installed (λ). The number of crashes before a treatment by itself is not a good estimate of π because of changes in safety that may result from changes in traffic volume, regression-to-the-mean, and trends in crash reporting and other factors. Instead, π is estimated from an EB procedure where information from the treatment site and a group of similar reference sites are combined. The following steps are used to estimate π:

1. Identify a reference group of untreated sites that is otherwise similar to the treatment group.
2. Use the reference group data to estimate safety performance functions (SPFs) (i.e., mathematical equations that predict crash frequency by type/severity as a function of traffic volumes and other site characteristics). Typically, SPFs are negative binomial regression models that are estimated using generalized linear modeling (GLM).
3. Calibrate annual multipliers (time-related factors) to account for temporal trends (e.g., variations in weather, demography, and crash reporting).
4. Use the SPFs and annual multipliers to compute the predicted number of crashes in each year of the before period for each treatment site.
5. Use the predicted number of crashes in the before period (from step 4) and the observed crashes in the before period at each treatment site to estimate the expected number of crashes in the before period at each site. This step applies the EB weighting scheme to adjust for possible bias due to regression-to-the-mean.
6. Estimate π (i.e., the expected number of crashes in the after period had the treatment not been implemented) as the product of the expected number of crashes in the before period (from step 5) and the ratio of the sum of annual SPF predictions for the after period divided by the sum of these predictions for the before period.
7. The estimate of π is then summed over all sites in a treatment group of interest and compared with the count of crashes during the after period.

The estimate of π and its variance are then used, along with the crash counts after the implementation of the treatment, to estimate the treatment effect.

Comparison Group Methodology

The comparison group method does not effectively account for regression-to-the-mean but can be effective in accounting for other non-treatment effects, such as those due to changes in traffic volume and other temporal trends. This method makes use of a comparison group that is untreated and for which an RSA was not conducted but is otherwise similar to the treatment sites. The data from the comparison group are used to adjust the before period crash count at the treated site to reflect the length of after period (relative to the before period), as well as other extraneous factors. Implicit in the use of a comparison group is the assumption that the comparison group is subject to extraneous factors affecting crashes in a way that is sufficiently similar to the treatment sites. This can be tested by tracking the yearly crash counts at treatment and comparison sites.

Similar to the EB method, the premise of the comparison group method is to estimate the expected number of crashes that would have occurred in the after period had the treatment not been implemented (π) and compare that with the number of reported crashes in the after period once the treatment is actually installed (λ). The departure from the EB method is that π is based on the observed crashes in the before period at the treatment site. This estimate could be biased if sites are selected for treatment because of a randomly high observed crash count (i.e., regression-to-the-mean).

General temporal trends are accounted for using crash data from the comparison group. Specifically, the ratio of crashes after treatment to before treatment is computed for the comparison sites and multiplied by the observed crashes before treatment at the treatment site. This is generally insufficient to account for changes in traffic volume from the before period to the after period. To properly account for traffic volume changes, another factor is applied to the result of the previous step. The comparison group is first used to develop an SPF, and the factor is computed as the ratio of predicted crashes in the after period divided by the predicted crashes in the before period for the treatment sites.

Naïve Methodology

The naïve (or simple) before-after method does not effectively account for regression-to-the-mean and is also less effective than the EB and comparison group methods in accounting for other non-treatment effects, such as those due to changes in traffic volume and other temporal trends. This method does not utilize a reference or comparison group and simply uses before and after data from the treatment sites.

Similar to the previous two methods, the Naïve before-after method estimates the expected number of crashes that would have occurred in the after period had the treatment not been implemented (π) and compares that with the number of reported crashes in the after period once the treatment is actually installed (λ). Similar to the comparison group method, π is based on the observed crashes in the before period at the treatment site, but there is no use of a comparison group to adjust this estimate. Again, this estimate could be biased if sites are selected for treatment based on a randomly high crash count (i.e., regression-to-the-mean).

The Naïve before-after method accounts for the length of the before and after periods and roughly adjusts for changes in traffic volumes. There is no account for other temporal trends that could change from the before to the after period. To account for traffic volume changes, a simple linear relationship between crashes and traffic volume is often assumed. In this case, the ratio of traffic volume in the after period to the before period would be calculated. The ratio of years after treatment to years before treatment is also computed to adjust for the number of years in each period. The observed crash count before treatment is then multiplied by the two ratios to estimate π.

 

< Previous

Table of Content

Next >