The rapid increase in the sophistication of vehicle automation demands development of evaluation protocols tuned to understanding driver-automation interaction. Driving simulators provide a safe and cost-efficient tool for studying driver-automation interaction, and this paper outlines general considerations for simulator-based evaluation protocols. Several challenges confront automation evaluation, including the limited utility of standard measures of driver performance (e.g., standard deviation of lane position), and the need to quantify underlying mental processes associated with situation awareness and trust. Implicitly or explicitly vehicle automation encourages drivers to disengage from driving and engage in other activities. Thus secondary tasks play an important role in both creating representative situations for automation use and misuse, as well as providing embedded measures of driver engagement. Latent hazards-hazards that exist in the road environment and merit driver attention, but do not materialize to require a driver response-have been used with great success for understanding the vulnerability of novice drivers. Latent hazards might provide a similarly useful index of driver attention to the road during periods where the automation is vulnerable to failure. With highly automated vehicles, latent hazards include potential roadway threats that might not be sensed by the automation and would require driver attention. This paper describes driving simulator scenarios used to operationalize automation-relevant latent hazards, secondary tasks tuned to index driver disengagement from the driving task, and measures that reflect safety margins rather than driving performance, such as drivers’ trust, situation awareness, and expected time to transition to manual control.
