Evaluating the Benefits of Emerging Automotive Technologies

How do you determine the effectiveness of a technology that does not yet exist, or even one that exists but is not yet in widespread use? Manufacturers, regulators, and consumers are all keenly interested in the potential benefits of emerging technologies, but a technology must be adopted before real-world data is available for robust, accurate benefits assessment.

For this reason, benefits analysis requires a holistic, incremental approach that starts with simplified measures. Different methods are appropriate at different stages of development, deployment, and adoption, with earlier studies requiring broad assumptions that can be updated or replaced as additional data becomes available.

In the case of automotive technology, analyses can ultimately incorporate results from real-world crash data and even telemetry. Continuous evaluation of new information is essential, and a variety of different types of expertise — from automotive engineering to statistics to human factors — is required to understand how they fit together to provide a consistent and accurate understanding over time.

From an Idea to Widespread Deployment: Lane Departure Warning Systems

To understand the challenges involved in evaluating the benefits of the technologies of tomorrow, we can look back to when today's technologies only existed on the drawing board.

Consider our evolving understanding of the potential impact of lane departure warning (LDW) systems on crash rates over the past three decades. LDW systems use cameras to detect the edges of the travel lane and alert the driver when a vehicle appears to be drifting out of the lane. This advanced driver assistance system (ADAS) feature is intended to help prevent road departure and centerline crossing accidents when the vehicle drifts laterally due to, for example, driver inattention or fatigue.

Early projections of the effectiveness of lane-keeping systems were made even before any such systems had been built. A 1996 National Highway Traffic Safety Administration (NHTSA) study evaluated potential benefits of various ADAS technologies, including LDW. At the time, no operational system existed for assessment, so the benefits calculation was performed using a simple mathematical formula that assumed an effectiveness, or crash prevention probability, for different relevant crash scenarios.

Combining effectiveness with frequencies for the crash scenarios themselves, this study estimated that an LDW system could reduce target crash scenarios by 75% and overall road departure crashes by 8%. However, this early bounding assessment assumed universal system installation and activation (100% adoption); system functionality under all weather, lighting, and driving conditions (100% availability); and correct functioning of the system at all times (100% reliability). These assumptions are implausible.

A decade later — after the first commercially available LDW system had been introduced in the United States in model year 200 5 — NHTSA published the results of a field operational test (FOT) conducted by the University of Michigan Transportation Research Institute (UMTRI) for a road departure warning system similar to the one envisioned in 1996. It assessed the driving behavior of 78 volunteers who used the study vehicles in place of their regular vehicles for four weeks.

The study was relatively small and did not observe any real road departure crashes, so safety benefits were estimated based on road departure conflicts not resulting in a crash. Incorporating the observed system availability and reliability rates, which accounted for factors such as the system frequently failing to function off freeways or in the rain, gave crash reduction estimates of 2 — 16% for relevant scenarios. However, these estimates assume that reductions in non-crash incidents will correspond well to reductions in crashes.

The reduction from 75% to 16% effectiveness is not solely due to changes in the study methodology. Benefits analysis studies like these generally refer to improvement in "relevant" or "target" crashes, and very different results may be observed depending on which crashes are included. A common challenge in comparing results between studies, including all of the studies described here, is that the quoted effectiveness numbers apply to different populations of crashes.

Field tests to assess risk reduction for rare events, such as road departure crashes, are challenging due to the large number of vehicle miles traveled required to observe multiple events. One way to acquire extensive information at a reasonable cost is to leverage simulation.

A 2010 report describes a collaboration between Volvo, Ford, and UMTRI using crash scenario modeling, including computational models for LDW performance and driver response, augmented by targeted experimental, FOT, and simulator data. Accounting for system availability and additional factors, such as driver noncompliance with alerts, resulted in an estimated effectiveness range of 13 — 32%. By model year 2018, more than 30% of new vehicles were equipped with some form of LDW. As a result, additional real-world driving data is now available for benefit assessments and projections.

A 2018 study by the Insurance Institute for Highway Safety (IIHS) estimated that vehicles equipped with LDW had a reduction of relevant crash types from as low as 1% up to 20%. The study used a statistical model applied to crash data for vehicles with optional LDW systems from six manufacturers and controlled for other features, such as driver characteristics or advanced headlight systems, that might also influence crash rates.

In understanding these relatively low effectiveness estimates, it is important to consider other factors such as drivers deactivating LDW systems due to false alerts, which could not be effectively assessed before widespread deployment.

Still an Incomplete Picture

Table 1 gives a brief description of key elements of these studies, including data sources and assumptions. Generally speaking, more relevant and sophisticated data has become available over time, and the number of broad, simplifying assumptions needed to understand the benefits has tended to decrease. However, none of these studies considered the same population of crashes to be relevant to LDW, which limits our ability to directly compare results, and none of these studies, individually, gives a complete picture of LDW effectiveness.

Experimental studies are generally run at small scales, simulations rely critically on assumptions, and large-scale retrospective studies must leverage statistical models to compensate for the absence of control over confounding factors. That said, all have consistently indicated that the prevention of all relevant crashes is unrealistic given the performance and use of current LDW technology. The broad ranges within and across estimates also highlight the uncertainty in these types of assessments.

Current estimates based on large-scale studies are not the last word in LDW effectiveness. This type of technology is still under development, with ongoing adjustments to LDW interfaces and lane-marking recognition systems. In addition, more sophisticated ADAS features, like technologies that can actively steer a car away from the edge of a lane without driver input, known as lane keeping assist (LKA) systems, are becoming more common and might be able to address situations that a driver alert system cannot.

Understanding the impact of these improvements will require the use of targeted experimental data and simulations until their adoption is widespread enough to see their impact in the national crash data.

How Exponent Can Help

When considering emerging technologies, it is critically important to capture benefits information using the best available data and methods, to understand the limitations of those methods, and to continue to update these estimates as the technology matures. With state-of-the-art testing facilities, Exponent's automotive consultants leverage expertise in national and state crash databases, technology development and deployment, advanced simulation, data sciences, and human factors to support benefits estimates at every stage of development, and our integrated approach ensures that as soon as new data are available they can be integrated with previous knowledge to give the clearest possible picture of the future.