Chapter 4 – Using Evaluation: The Focus is Results
The preceding chapters introduced the benefits of evaluation, tips on evaluation planning, and methods for evaluating both SHSP process and performance. The purpose of this chapter is to provide information and encouragement for using evaluation results to improve the SHSP process and performance.
Introduction
The EPM introduction explained the basic purpose of evaluation as keeping the SHSP process open to feedback, change, and improvement. Assume for a moment that recommendations from the previous chapters have been implemented and accomplished. The next steps are to interpret the data and use them to improve the SHSP.
Interpret the Data
Source: Cambridge Systematics, Inc.
It is essential to interpret, document, and share SHSP evaluation results with managers and stakeholders. It helps them: understand organizational strengths and challenges; gauge progress in meeting goals and objectives; and make informed decisions. The interpretation depends on the data collected and the questions rasied. No singular list of questions is applicable to all situations. Rather, example questions based on possible process and performance results are provided below.
SHSP Process
Suppose the analysis of process evaluation data reveals that, over time, the high-level decision-makers who were originally involved no longer attend the steering committee meetings and have delegated the task to staff. Answers to the following questions would help interpret the data:
- Do the staff members have access to leadership?
- Can they make decisions on behalf of leadership?
- Are the staff members able to secure resources when needed?
- Is enthusiasm for SHSP implementation and evaluation dwindling?
Evaluation in Action
Alaska Uses Evaluation to Educate the Public on Traffic Safety
Alaska used evaluation results to announce the update of their Strategic Traffic Safety Plan (STSP) at a media event attended by the State’s top traffic safety officials. Hosted by the Deputy Commissioner of the Department of Transportation and Public Facilities, the event included presentations from a U.S. Congressman; Commissioners for the Departments of Public Safety, Health and Social Services, and Administration; and a representative from the Department of Corrections. Evaluation data were used to highlight the State’s traffic safety issues and to celebrate successes such as declines in unrestrained, alcohol-related, and speeding fatalities. The officials applauded the declining numbers but said one death on Alaska’s highways is too many. They urged everyone to embrace the STSP’s goal of “Toward Zero Deaths.”
Depending on the answers to these and other questions, the current steering committee could determine that delegating the steering committee participation to the staff level does not hinder SHSP process and performance. On the contrary, it is improved because the current members have fewer competing priorities and duties and are able to devote more time and attention to the work. On the other hand, if lack of leadership presence has resulted in fewer resources and less decision-making authority, methods would need to be devised to reengage the leaders.
SHSP Performance
Analysis of performance evaluation data may reveal the number of fatalities and serious injuries in some emphasis areas are not decreasing as planned.
Again, the solutions will vary, but any of the following challenges could be underlying the failure to make progress:
- Data deficiencies, e.g., inability to identify problem locations;
- Implementing ineffective (unproven) strategies;
- Inadequate resources to implement the countermeasures;
- Lack of sufficient authority among those charged with implementation;
- Inadequate expertise or understanding of how to implement the strategies; or
- Lack of the political support necessary to change public attitudes.
Identifying and implementing solutions begins with correctly defining the problem. Solutions may then become obvious, but first the data need to be interpreted sufficiently.
For example, if an SHSP goal or objective is not met, the results may suggest a strategy is ineffective; however, in some cases the strategies may not have been implemented as intended. In this case, a State would need to determine if new strategies are needed or if implementation issues need to be addressed.
Apply the Results
Evaluation results must be reviewed to identify and document ways to improve SHSP process and performance. These “lessons learned” should be widely shared along with the evaluation results. Of course, once the data are interpreted and shared, a commitment to use the results is necessary.
Operationalize the Findings
Evaluation in Action
Missouri Uses Evaluation Results to Inform Stakeholders and Examine Emphasis Areas
Data are critical to the success of the Missouri SHSP evaluation process, which is why the State determined safety stakeholders should regularly access up-to-date data. The State compiles and distributes statewide, regional, and county fatality numbers to all safety coalition members weekly, which allows comparison to previous time periods. To compile the information, the State uses a standardized tracking methodology, which enables consistent longitudinal analysis. The data include information on both behavior and infrastructure, e.g., areas of the State with low safety belt use or infrastructure improvements by location and installation date to enable before-and-after evaluations. Distributing the data keeps stakeholders involved in the SHSP process and regularly reminds them of the overall goal to save lives and prevent injuries. The data also helps regions and localities examine their data closely to identify the most pressing highway safety problems.
Evaluation results should be used to enhance the SHSP process, improve performance, inform managers and stakeholders, and facilitate decision-making.
States can use evaluation results to accomplish the following purposes, among others. These will be particularly important when updating the SHSP:
- Verify SHSP strategies and action plans are being implemented as intended;
- Validate the expected effectiveness of SHSP processes, strategies, and programs;
- Identify effective processes, strategies, and programs for replication;
- Identify weaknesses in the SHSP strategies and actions, such as failure to implement strategies, as well as strategies not achieving the intended results;
- Improve SHSP implementation;
- Direct resources to areas with the highest probability of improving safety;
- Inform elected officials, the media, and the public about the SHSP’s impact;
- Identify potential leaders and partners across disciplines and modes; and
- Generate public understanding of the SHSP and recruit participation and support.
Share the Information
States should establish a feedback loop to ensure evaluation results are studied, used, and incorporated in the SHSP process. Evaluation results should be shared with decision-makers, SHSP committees and teams at all levels, safety stakeholders, and as appropriate with the public and the media. Information sharing informs partners and stake- holders about the SHSP process and performance. SHSP stakeholders should be aware of successful, as well as deficient programs as soon as the informa- tion is available, reviewed, and validated. This enables leaders and managers to address challenges; implement course corrections; educate stakeholders about the SHSP process, programs, and activities; inform the public to increase understanding and support for the SHSP; and inform elected and appointed officials so they consider additional funding and support for safety efforts.
Sharing results helps sustain partnerships and momentum and can help persuade decision-makers to support evaluation as an integral part of the SHSP process. As awareness about transportation safety issues and programs increases, further support for the SHSP is generated. Results are typically shared through press releases, news media events, brochures, formal reports, newsletters, presentations, and other methods.
Many States hold Safety Summits or conferences to share information and generate enthusiasm for SHSP implementation. Sessions could highlight evaluation results, especially noteworthy or exemplary practices that encourage others to adopt similar strategies in their communities or organizations. Sharing less successful results is also beneficial, especially if this results in brainstorming, consensus building, and solutions development.
Using Evaluation Checklist
The following checklist is designed to support the use of evaluation results. If an action on the checklist is in progress or completed, the State is well on the way to using evaluation results to improve the SHSP process and performance.
- Evaluation results have been interpreted and documented.
- Evaluation results were reviewed to identify lessons learned.
- Lessons learned have been used to improve SHSP process and performance.
- SHSP stakeholders are made aware of both successful and unsuccessful programs and strategies as soon as sufficient information is available.
- Evaluation results are being used to increase public understanding of SHSP programs and strategies.
- Evaluation results are being used to help inform elected and appointed officials so they might support increased funding and resources for safety programs and strategies.
- Evaluation results are being used to identify additional safety leaders and partners.
- Evaluation results that identify gaps and weaknesses in SHSP process or performance are being addressed through follow-up actions.
- Evaluation results are used to direct resources to areas with the highest probability of improving safety.
