Automated Driving Systems (ADS) make the driving experience safer, more efficient, and comfortable by performing complex maneuvers, preempting potential risky situations, or taking over the driver’s tasks in critical situations. Innovation acceptance research for Advanced Driver Assistance Systems (ADAS) illustrates the increasing demand for safety and comfort as the two prime movers of the ADAS market. Since ADAS technologies have significant impact on human lives, extensive testing and validation throughout the design process is indispensable. Due to complexity of systems, cost of testing, and safety of test engineers, a significant chunk of ADAS calibration and validation needs to be done virtually. Although simulation-based verification and validation (V&V) is not new, the test descriptions, modeling and simulation framework are not yet well established. Off-the-shelf software tools have different architectures, simulation procedures, data standardizations, and tradeoffs. The underlying question remains- “How to ensure that a simulated test scenario actually tested or validated the ADAS algorithm?” This paper attempts to better understand the basic requirements to successfully test common ADAS algorithms by describing a generalized structure of simulation framework essential to successfully run the safety tests. The study also emphasizes how variability across different simulators may cause discrepancies in the results. The scenarios are built in two different simulators which have unique features, methods and assumptions that must be well-understood for the results to be proven valid. Finally, the essential features of simulators are documented to understand the effect of simulator specific scenario parameters on the success of test simulations via test-of-tests.