Data Collection

Crash Data Improvement Program (CDIP) Final Report

SUMMARY, KEY FINDINGS AND FUTURE CONSIDERATIONS

The CDIP is completing its fifth year of operation; third year with the current contractor. Over that time it has been modified and updated in response to feedback from the participants. In particular, States asked for more relevant examples. The CDIP contents and process were revised to include the State's own data in the Workshop and Technical Transfer sessions, where possible. This change is significant because it also helped the TAT better understand the technical barriers facing analysts who would be charged with calculating the measures of data quality performance recommended in the CDIP reports. Only one State among the ten opted to conduct their own data analyses–in that State's CDIP, the TAT used the results tables provided by the State as content in the Workshop.

Two States have requested follow-up assistance through their FHWA Division Office and NHTSA Regional Office representatives. In both cases, the need was for help in combining multiple strategic plans into a single comprehensive document. One of the States opted to create a business plan that showed the linkages among several projects designed to improve the crash data. This helped the State identify task dependencies that might delay high-priority projects because a crucial activity in a different project was not completed in time. The other State needed help combining its Traffic Records Strategic Plan with its SHSP to produce one document that upper-level management in the affected agencies could review and endorse.

This effort went beyond the CDIP in that it included all of the traffic records components; however, the single largest category of projects in both plans was that related to crash records improvement.

Other States have said that one of the reasons they scheduled the CDIP was because they wanted to integrate CDIP findings in their Strategic Plan for Traffic Records and their SHSP. The other thing that has happened in the aftermath of a CDIP is that the States see the need to integrate their plans more thoroughly–that projects they've been pursuing on a separate course may in fact interact in ways that can and should be part of the plan.

The CDIP has evolved into a planning tool for States. As such, it also has taken on an important role in supplementing the Traffic Records Assessments conducted by NHTSA. In recognition of this fact, FHWA and NHTSA have agreed to transition the CDIP from FHWA to NHTSA. Federal Fiscal Year (FFY) 2013 is a transition year in which both administrations have worked cooperatively to manage the program. In FFY 2014, NHTSA will take full control of the CDIP. In anticipation of that event, the contract team worked with NHTSA and FHWA to modify the CDIP guide, the pre-site questionnaire, the pre-site visit information request, and the contents of the CDIP Workshop presentations. Most of the changes were made to more closely align the CDIP terminology with the wording used in the NHTSA Traffic Records Advisory and Assessment. In addition, the pre-site visit information request and questionnaire were cross-referenced to show States which items they may have already provided or answered as part of their Assessment. In the future, this will be a useful feature because it is expected that the CDIP will become part of NHTSA's follow-up with States requesting help in response to recommendations in their Traffic Records Assessment. States can request help, in the form of Go Teams, for a variety of traffic records improvement projects. The CDIP TAT will be one example of a NHTSA Go Team.

Throughout the period of 2010 through 2013, the CDIP TAT advised States to establish formal, comprehensive data quality management programs. The description of what such a program might look like has become a shared item among FHWA and NHTSA. The 2011 revision of the crash section of the NHTSA Traffic Records Advisory includes the same description of a formal, comprehensive data quality management program as is presented in the CDIP reports. That description was also used in the last two-to-three years of Traffic Records Assessments (i.e., as the 2011 revision was being developed, but before it was first used in a State Traffic Records Assessment). As a result of this coordination, there is now one well-documented list of what a State's crash data quality management program should include. The components are:

• Automated edit checks and validation rules that ensure entered data fall within the range of acceptable values and is logically consistent between other fields. Edit checks are applied when data are added to the record. Many systems have a two-tiered error classification system, distinguishing critical errors that must be corrected before submission and non-critical error warnings that may be overridden.
• Limited State-level correction authority is granted to quality control staff working with the statewide crash database to amend obvious errors and omissions without returning the report to the originating officer. Obvious errors include minor misspellings, location corrections, and directional values. Obvious omissions include missing values that can easily be obtained from the narrative or diagram. States may opt to grant no editing authority to quality control staff and still meet this description–the point is that the authority must be limited.
• Processes for returning rejected crash reports are in place to ensure the efficient transmission of rejected reports between the statewide data system and the originating officer as well as tracking the corrected report's submission.
• Performance measures are tailored to the needs of data managers and address the concerns of data users. Measures can be aggregated from collectors, users, and the State TRCC. The crash data should be timely, accurate, complete, uniform, integrated, and accessible. These attributes are tracked using State-established quality control measures. The State is encouraged to develop additional measures that address their specific needs.
• Numeric goals–or performance metrics–for each performance measure are established and regularly updated by the State in consultation with users via the TRCC.
• Performance reporting provides specific feedback to each LEA on the timeliness, accuracy, and completeness of their submissions to the statewide crash database relative to applicable State standards.
• High-frequency errors are used to generate new training content and data collection manuals, update the validation rules, and prompt form revisions.
• Quality control reviews comparing the narrative, diagram, and coded report contents are considered part of the statewide crash records system's data acceptance process.
• Independent sample-based audits are conducted periodically for crash reports and related database contents. A random sample of reports is selected for review. The resulting reviews are also used to generate new training content and data collection manuals, update the validation rules, and prompt form revisions. At a minimum, these audits occur on an annual basis.
• Periodic comparative and trend analyses are used to identify unexplained differences in the data across years and jurisdictions. At a minimum, these analyses occur on an annual basis.
• Data quality feedback from key users is regularly communicated to data collectors and data managers. This feedback will include corrections to existing records as well and comments relating to frequently occurring errors. Data managers disseminate this information to law enforcement officers as appropriate.
• Data quality management reports are provided to Department Administrator(s) and TRCC for regular review. The TRCC used the reports to identify problems and develop countermeasures.

Eight of the ten CDIP implementations discussed in this report included the finding that the State lacked a "formal, comprehensive data quality management program." This is not to imply that the other two States had such a program in its entirety. Rather, the CDIP TAT found, in those two States, that the existing quality control program was functioning sufficiently well that the recommended quality control improvements (especially in the area of measurement) could be added to the existing management program.

It is likely, based on experiences in CDIP and in the Traffic Records Assessments over the most recent five-year period, that very few States (likely less than 20 percent), have a crash data quality management program. Most will have heard that their existing data quality management program lacks specific components from the list presented above. It is also likely that a majority of States will need help to establish such a program. In the CDIP reports, the TAT very often recommends that the State proceed gradually in establishing and formalizing a data quality management program. This is important because (a) States should establish the program that makes the most sense for their own situation, and (b) it is costly and resource-intensive to set up such a program when most of the effort will be manual in nature. The better course is to work toward the more formal data quality management program by including it in the Traffic Records Strategic Plan as a project (or series of projects) and to link those projects to specific upgrades to the crash data management system (software and analytic capabilities) over time. Automating the components of a data quality management program–especially the components related to calculating data quality performance measures–helps to lower costs and increases the likelihood that the State will be able to maintain the effort in the long term.

As noted earlier in this section, States are also very likely to need help with strategic planning. The CDIP includes a review of the documents the State provides, including their most recent Traffic Records Strategic Plan update and the SHSP. In some States, the two plans are not coordinated. The CDIP has also uncovered that some States do not include sufficient details in their plans to fully describe the process they will use to reach key goals–including the goal to increase electronic submission of crash reports by LEAs. If a State lacks knowledge of the technological capabilities of the LEAs, for example, it cannot effectively plan the future of electronic submissions from those agencies. Where States have detailed, up-to-date information on the capabilities of LEAs to collect and submit crash data electronically, staff can also determine how best to meet the needs for interfaces, data submission guidelines, a certification process, and test procedures. Several States appear to need some assistance in setting up an effective process, especially if they are trying to accommodate multiple different crash data collection or records management systems in use among LEAs.

Perhaps the most critical need seen among the CDIP States is for a single official repository of crash data. Many States are faced with a situation in which it is easier and cheaper for key users of the crash data to simply take a copy (a data extract) for their own use, largely relegating the official crash data system to the role of repository of the source data. Users in this situation will make improvements to the copy of the data that they maintain. Unfortunately, this "cheaper, quicker" method has unrecognized costs associated with its long-term use. Most notably, States in this situation often find that the highest quality crash data is not in the official crash data repository, but in some other system maintained by a key user agency. Worse, there may not be a single "best" version of the data. The corrections made in a State DOT are different from those that would be made in support of an injury surveillance program, for example. The highest quality data would be achieved only if the corrections applied by all key users were available for analysis by others. States that can transition their legacy centralized crash databases to serve multiple users with complete and corrected data will ultimately save money by reducing redundancy and sharing analytic knowledge and resources.

Although this benefit has not yet been measured in any State, the reduction in redundant databases and the improved communications implicit in a formal data quality management program should generate measurable savings and data quality improvements.

In the ideal centralized system, the crash database would have a record of the data as it was originally submitted along with a log of all changes that have been made to the data. That change log (or transaction table) can be used to roll-forward or roll-back a crash record to access any desired set of changes, from none at all to every change of a particular type to every change made for any purpose. This type of system is ideal for automating the calculation of crash data quality measurements of accuracy and completeness. Once the relevant reports are created, the crash data managers could obtain at any time, and for any time period, a report that shows (a) the most frequent errors, (b) the LEAs making the errors, and (c) the most frequent corrections to specific errors. This information not only describes the quality of the data in the crash system, it also supports several of the components of a formal, comprehensive crash data quality management program. With this information, a State could update its crash report form and data collection manual, add specific examples of likely errors to training and messages sent to law enforcement in general, and provide specific feedback to individual LEAs on their own most frequent errors. In addition, some of the errors can be eliminated through imposition of new edit checks and validation rules so that field data collection software and the data intake process for the statewide database can catch the errors before they end up in the system.

It should be clear that States may not achieve this ideal system quickly or without cost. The CDIP reports include advice to States that they should add these capabilities as they are redesigning or upgrading their crash records systems. At that time, the data quality support features can be added to the system for a relatively small incremental cost. Grafting these capabilities onto an existing system, conversely, is likely to be difficult and expensive. Follow-up with CDIP States in three to five years after they received the recommendations is likely to show substantial progress in formalizing data quality management practices, especially in those States that have upgraded their crash data management systems.