U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
|< Previous||Table of Contents||Next >|
The purpose of this step is to describe how to evaluate the impact of the treatments that have been implemented in terms of crash frequency or severity. A reliable assessment of the effectiveness of safety countermeasures cannot be made immediately after implementation. Some time needs to pass, often two to three years, before enough data can be collected to determine how many crashes, serious injuries, and fatalities have occurred since implementation of the countermeasure and then compare it with the same types of data from before implementation.
This step should not be overlooked. Evaluation provides information that can help agencies decide whether or not the investment has reduced crash frequency or severity. Evaluation also can help agency staff demonstrate the value of the program to community leaders and the general public.
The level of effort required to conduct an evaluation depends on the resources available and the method chosen. Some methods are:
At the most basic level, evaluation is a “before and after” comparison. This means that some sort of “before” or “baseline” data must be available against which the “after” data can be compared. This baseline data is often available as part of the information used to select the site in the first place. To demonstrate the benefits of safety investments, the agency should track before and after performance measures, analyze trends, and conduct selective benefit/cost analysis. The Highway Safety Improvement Program manual provides detailed information about how to do this. Please see the Resources at the end of this section.
Comparative Assessment – Treatment at One Site. Figure 17 illustrates a comparative analysis of before and after conditions conducted after a treatment has been implemented at a single site.
Charts A, B, and C summarize crash counts, average daily traffic, and target crash type crash counts for the three years before and after implementing the project. Charts D and E show the relative (percent) and absolute (number) changes in traffic volumes and crashes after implementation. Visually comparing these characteristics side by side allows a qualitative assessment of the treatment’s effectiveness. A comparative assessment considers crash counts, crash type and severity and traffic volume to develop a qualitative assessment of the changes from before to after deploying a treatment.
Source: Oregon Department of Transportation, Evaluating the Effectiveness of the Safety Investment Program (SIP) Policies for Oregon.
Simple Before/After Crash Analysis. A simple before/after crash analysis compares the number of crashes that occurred during the three-year period before a treatment is installed to the number of crashes that occurred in the three-year period after the treatment is installed (see Table 11). A simple before/after crash analysis is focused only on crash counts for the purposes of estimating the quantitative benefits of deploying a treatment.
|Total Crashes at Site||22||16||27%|
This type of analysis is an appealing way to estimate the quantitative safety benefits of a treatment because it is easy to complete, requires little data, and is easy to explain. But there are some important limitations to it. The main problem with it stems from the underlying assumption that the number of crashes experienced before implementing the countermeasure is a good estimate of the number of crashes that would be expected in the future assuming the countermeasure had not been implemented. However, this is not always the case because traffic volumes or surrounding land uses may change or levels of enforcement may change. Due to this and other limitations, it cannot be relied upon solely to evaluate effectiveness.
Rigorous Before/After Crash Analysis. To address the weaknesses inherent in simple before/after crash analysis, methods that utilize more sophisticated statistical tools are used. These include before/after analysis with comparison groups, empirical Bayes (EB) analysis, full Bayes studies, and cross-sectional studies. These methods should be used whenever an agency wants to develop a reliable quantitative estimate the effectiveness of a treatment (to develop a Crash Modification Factor (CMF) for example). The Highway Safety Manual and “Recommended Protocols for Developing Crash Modification Factors” available on the Crash Modification Factor Clearinghouse web site provide information and guidance on using these methodologies. Please see the Resources at the end of this section.
Empirical Bayes Analysis – What is it?
The Empirical Bayes (EB) method is a statistical method that combines the observed crash frequency with the predicted crash frequency using SPFs (see below) to calculate expected crash frequency. By combining crash history and predicted crash frequency, a better estimate of expected crash frequency can be made.
SPF (safety performance function) represents the change in mean crash frequency as ADT (or other exposure measure) increases or decreases.
The EB method overcomes regression-to-the-mean.
Effectiveness evaluations are typically conducted two to three years after the treatment has been in place. There are three primary reasons for this:
Chapter 6 of the Highway Safety Improvement Program (HSIP) manual discusses various safety evaluation methods and provides calculation example for each method. The manual’s Table 6.1 compares data and inputs required for these methods.
The report is FHWA Report Number: FHWA-SA-09-029.
This FHWA guide provides guidance for users to develop crash modification factors (CMF). Table 12 in the guide provides users with a comparison of evaluation study designs. The guide’s Chapter 4 provides a flowchart for users to select preferred evaluation design based on data availability and project.
The report is FHWA Report Number: FHWA-SA-10-032.
Volume 1, Chapter 9 of the HSM documents and discusses the various methods for evaluating the effectiveness of a treatment, a set of treatments, an individual project, or a group of similar projects after safety improvements have been implemented. The chapter also highlights which methods are appropriate for assessing safety effectiveness in specific situations, and provides step-by-step procedures for conducting safety effectiveness evaluations.
This is a NHI course that presents users with a description of safety effectiveness evaluation, an overview of fundamentals for performing safety effectiveness evaluation, and information about the importance of safety effectiveness evaluation in the context of the HSIP. Users are provided example before and after studies, and learn about the data needs for each methodology. The course lasts five hours and is partially instructor led and partially web based.
More course information is available on the National Highway Institute’s web site.
You may need Adobe® Reader® to view the PDFs on this page.
|< Previous||Table of Contents||Next >|