U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
FacebookYouTubeTwitterFlickrLinkedIn

Safety

eSubscribe
eSubscribe Envelope

FHWA Home / Safety / Local and Rural Road / Training

Improving Safety on Rural Local and Tribal Roads — Safety Toolkit

< Previous Table of Contents Next >

Step 7. Evaluate Effectiveness

Overview

The purpose of this step is to describe how to evaluate the impact of the treatments that have been implemented in terms of crash frequency or severity. A reliable assessment of the effectiveness of safety countermeasures cannot be made immediately after implementation. Some time needs to pass, often two to three years, before enough data can be collected to determine how many crashes, serious injuries, and fatalities have occurred since implementation of the countermeasure and then compare it with the same types of data from before implementation.

This step should not be overlooked. Evaluation provides information that can help agencies decide whether or not the investment has reduced crash frequency or severity. Evaluation also can help agency staff demonstrate the value of the program to community leaders and the general public.

The level of effort required to conduct an evaluation depends on the resources available and the method chosen. Some methods are:

At the most basic level, evaluation is a “before and after” comparison. This means that some sort of “before” or “baseline” data must be available against which the “after” data can be compared. This baseline data is often available as part of the information used to select the site in the first place. To demonstrate the benefits of safety investments, the agency should track before and after performance measures, analyze trends, and conduct selective benefit/cost analysis. The Highway Safety Improvement Program manual provides detailed information about how to do this. Please see the Resources at the end of this section.

Example

Comparative Assessment – Treatment at One Site. Figure 17 illustrates a comparative analysis of before and after conditions conducted after a treatment has been implemented at a single site.

Charts A, B, and C summarize crash counts, average daily traffic, and target crash type crash counts for the three years before and after implementing the project. Charts D and E show the relative (percent) and absolute (number) changes in traffic volumes and crashes after implementation. Visually comparing these characteristics side by side allows a qualitative assessment of the treatment’s effectiveness. A comparative assessment considers crash counts, crash type and severity and traffic volume to develop a qualitative assessment of the changes from before to after deploying a treatment.

Figure 17. Example of Comparative Analysis of Before and After Conditions

Figure 17 shows a graphic representation of crashes by type and year, average daily traffic volume, target crash frequency by year, percent and count in change in traffic volume, target crashes, and crash severity. Collectively these graphs can be used to evaluate the change in site conditions after implementing a treatment.

Source: Oregon Department of Transportation, Evaluating the Effectiveness of the Safety Investment Program (SIP) Policies for Oregon.

Simple Before/After Crash Analysis. A simple before/after crash analysis compares the number of crashes that occurred during the three-year period before a treatment is installed to the number of crashes that occurred in the three-year period after the treatment is installed (see Table 11). A simple before/after crash analysis is focused only on crash counts for the purposes of estimating the quantitative benefits of deploying a treatment.

Table 11. Example Simple Before/After Analysis
  2006-2008 2009-2011 Percent Reduction
Total Crashes at Site 22 16 27%

This type of analysis is an appealing way to estimate the quantitative safety benefits of a treatment because it is easy to complete, requires little data, and is easy to explain. But there are some important limitations to it. The main problem with it stems from the underlying assumption that the number of crashes experienced before implementing the countermeasure is a good estimate of the number of crashes that would be expected in the future assuming the countermeasure had not been implemented. However, this is not always the case because traffic volumes or surrounding land uses may change or levels of enforcement may change. Due to this and other limitations, it cannot be relied upon solely to evaluate effectiveness.

Rigorous Before/After Crash Analysis. To address the weaknesses inherent in simple before/after crash analysis, methods that utilize more sophisticated statistical tools are used. These include before/after analysis with comparison groups, empirical Bayes (EB) analysis, full Bayes studies, and cross-sectional studies. These methods should be used whenever an agency wants to develop a reliable quantitative estimate the effectiveness of a treatment (to develop a Crash Modification Factor (CMF) for example). The Highway Safety Manual and “Recommended Protocols for Developing Crash Modification Factors” available on the Crash Modification Factor Clearinghouse web site provide information and guidance on using these methodologies. Please see the Resources at the end of this section.

Empirical Bayes Analysis – What is it?

The Empirical Bayes (EB) method is a statistical method that combines the observed crash frequency with the predicted crash frequency using SPFs (see below) to calculate expected crash frequency. By combining crash history and predicted crash frequency, a better estimate of expected crash frequency can be made.

SPF (safety performance function) represents the change in mean crash frequency as ADT (or other exposure measure) increases or decreases.

The EB method overcomes regression-to-the-mean.

Application

Effectiveness evaluations are typically conducted two to three years after the treatment has been in place. There are three primary reasons for this:

  1. It takes time for enough data to accumulate so that meaningful results can be obtained.
  2. Road users, who may be quite familiar with the roadway environment “pre-treatment,” will often modify their driving behavior “post-treatment” until they become familiar with the new environment. Because this interim period does not reflect the long-term impact of the treatment, it is important to wait until drivers are used to the new roadway environment before evaluating effectiveness.
  3. In contrast, if the evaluation is conducted too many years after the treatment is implemented, conditions such as traffic volumes and land uses around the site may have changed to such a degree that any observed change in crash frequency or crash severity may not be attributable only to the treatment.

Resources For Step 7: Evaluating Effectiveness

Highway Safety Improvements Program Manual

Chapter 6 of the Highway Safety Improvement Program (HSIP) manual discusses various safety evaluation methods and provides calculation example for each method. The manual’s Table 6.1 compares data and inputs required for these methods.

The report is FHWA Report Number: FHWA-SA-09-029.

A Guide to Developing Quality Crash Modification Factors

Cover of Federal Highway Administration document: A Guide to Developing Quality Crash Modification Factors.

Source: FHWA.

This FHWA guide provides guidance for users to develop crash modification factors (CMF). Table 12 in the guide provides users with a comparison of evaluation study designs. The guide’s Chapter 4 provides a flowchart for users to select preferred evaluation design based on data availability and project.

The report is FHWA Report Number: FHWA-SA-10-032.

Highway Safety Manual

Volume 1, Chapter 9 of the HSM documents and discusses the various methods for evaluating the effectiveness of a treatment, a set of treatments, an individual project, or a group of similar projects after safety improvements have been implemented. The chapter also highlights which methods are appropriate for assessing safety effectiveness in specific situations, and provides step-by-step procedures for conducting safety effectiveness evaluations.

National Highway Institute (NHI) Highway Safety Improvement Program (HSIP) Project Evaluation

This is a NHI course that presents users with a description of safety effectiveness evaluation, an overview of fundamentals for performing safety effectiveness evaluation, and information about the importance of safety effectiveness evaluation in the context of the HSIP. Users are provided example before and after studies, and learn about the data needs for each methodology. The course lasts five hours and is partially instructor led and partially web based.

More course information is available on the National Highway Institute’s web site.

You may need Adobe® Reader® to view the PDFs on this page.

Return to top

< Previous Table of Contents Next >
Page last modified on October 4, 2013.
Safe Roads for a Safer Future - Investment in roadway safety saves lives
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000