Handbook of Driving Simulation for Engineering, Medicine, and Psychology

Simulator Validity: Behaviors Observed on the Simulator and on the Road

Authors
Nadia Mullen, Lakehead University
Judith Charlton, Monash University
Anna Devlin, Monash University
Michel Bédard, Lakehead University

Abstract
The Problem. Driving simulators offer a safe, convenient alternative to measuring driving performance on-road. However, the results of simulator studies may not generalize to driving in the real world if the simulator lacks behavioral validity. Behavioral validity refers to the extent to which the simulator elicits the same driving behaviors that occur when driving in the real world. Role of Driving Simulators. Validation is important to generate and maintain simulator use, acceptance, and credibility, and is vital when simulator performance influences real-world outcomes, such as road or vehicle design, or whether drivers retain their license. Key Results of Driving Simulator Studies. A review of studies evaluating the behavioral validity of simulators showed that simulators provide a valid tool for assessing a variety of driving performance measures such as speed, lateral position, brake onset, divided attention, and risky traffic behaviors. Simulators also appear sensitive to age-related changes in driving performance and cognition. Measures for which simulators do not appear valid are discussed, in addition to factors influencing validity, such as driving ability. Overall, the evidence reviewed in this chapter indicates that simulator driving behavior approximates (relative validity), but does not exactly replicate (absolute validity), on-road driving behavior. This is sufficient for the majority of research, training, and assessment purposes for which simulators are used. However, where absolute values are required, on-road measures will generally be necessary. Scenarios and Dependent Variables. Validation studies involve consideration of factors such as the research question, task conditions, and dependent measures, each of which can affect validity. We discuss these methodological considerations, as well as statistical techniques used to establish validity. Platform Specificity and Equipment Limitations. Assumptions about driving simulator validity are critically dependent on the specific experimental conditions under which the driving behaviors are compared. Variations across simulator equipment, software, and environment may affect the generalizability of validation results. Therefore, each simulator should be validated for its ability to measure the driving behavior of the cohort for which it is to be used.

Keywords
Simulator Validity, Behavioral Validity, Dependent Measures, Real-world Driving Outcomes

Key Points
• Simulators need to be validated to ensure the results of simulator studies generalize to driving in the real world. Behavioral validity refers to the extent to which the simulator induces the same driving behaviors that occur when driving in the real world. Behavioral validity has been further defined in terms of absolute and relative validity. While absolute validity requires that the two driving environments produce the same numerical values, relative validity is established when the differences between the two environments are in the same direction, and of the same or similar magnitude. Absolute validity is rarely established in driving simulator studies, but establishing relative validity is both necessary and sufficient for simulators to be useful research tools in most, though not all, cases.

• Methodological considerations for validation studies include factors such as the nature of the research question, task conditions, and dependent measures. Participant characteristics, simulator equipment, and how driving behaviors are measured in the simulator and real world can also affect validity. Statistical techniques used to establish validity include analysis of variance, correlation, and descriptive analyses of observed measures (e.g., speed profiles) of behavior measured on-road and during simulated drives.

• Simulators appear to provide a valid tool for assessing a variety of driving performance measures including speed, lateral position, brake onset, divided attention, and risky traffic behaviors. The majority of validated measures show relative validity but fail to meet requirements for absolute validity. Simulators do not appear valid for measures of braking force or for assessing the effects of some traffic control methods. Simulators can also induce similar physiological responses to those expected in the real world. Simulators appear sensitive to age-related changes in real-world driving performance and cognition, and can identify older drivers at risk of future traffic violations.

• Future research should investigate the variability of data in addition to measures of central tendency and methods to reduce the occurrence of simulator discomfort, because these factors can affect validity conclusions. There is also a need to establish the reliability of using driving simulators with specific cohorts, such as drivers with specific medical conditions; research to date has generally established validity with healthy cohorts. Regression modeling offers a useful approach to exploring the question of validation and the extent to which variables measured in the simulator predict real-world driving.

• Most studies support the use of simulators, finding that simulator driving behavior approximates (relative validity), but does not exactly replicate (absolute validity), on-road driving behavior. This is sufficient for the majority of research, training, and assessment purposes for which simulators are used. However, simulator users should remain aware that simulators do not always provide an accurate picture of on-road driving behavior, and validity should not be assumed; each simulator set-up should be validated for its ability to measure the driving behavior of the cohort for which it is to be used. Furthermore, where absolute values are required, on-road measures will generally be necessary.

Key Readings
Blaauw, G. J. (1982). Driving experience and task demands in simulator and instrumented car: A validation study. Human Factors, 24, 473–486.

Blana, E. (1996). Driving simulator validation studies: A literature review. Institute of Transport Studies, University of Leeds, UK.

Godley, S. T., Triggs, T. J., and Fildes, B. N. (2002). Driving simulator validation for speed research. Accident Analysis and Prevention, 34, 589–600.

Kaptein, N. A., Theeuwes, J., and van der Horst, R. (1996). Driving simulator validity: Some considerations. Transportation Research Record, 1550, 30–36.

Törnros, J. (1998). Driving behavior in a real and a simulated road tunnel – A validation study. Accident Analysis and Prevention, 30, 497–503.