This content is not included in your SAE MOBILUS subscription, or you are not logged in.
VoGe: A Voice and Gesture System for Interacting with Autonomous Cars
ISSN: 0148-7191, e-ISSN: 2688-3627
Published March 28, 2017 by SAE International in United States
Annotation ability available
In the next 20 years fully autonomous vehicles are expected to be in the market. The advance on their development is creating paradigm shifts on different automotive related research areas. Vehicle interiors design and human vehicle interaction are evolving to enable interaction flexibility inside the cars. However, most of today’s vehicle manufacturers’ autonomous car concepts maintain the steering wheel as a control element. While this approach allows the driver to take over the vehicle route if needed, it causes a constraint in the previously mentioned interaction flexibility. Other approaches, such as the one proposed by Google, enable interaction flexibility by removing the steering wheel and accelerator and brake pedals. However, this prevents the users to take control over the vehicle route if needed, not allowing them to make on-route spontaneous decisions, such as stopping at a specific point of interest.
This article presents VoGe, a human vehicle interaction system based on voice and pointing gestures that enables the user making spontaneous decisions over the route and communicate them to the car, while maintaining the interaction flexibility. We describe the system’s design and its main functions. In addition, a first proof of concept based on a robotic platform and a driving simulator is presented. This showcased the main functions of VoGe related to commanding an autonomous vehicle by using voice and pointing gestures. Furthermore, it helped to identify the main challenges that our team’s ongoing research will focus on; such as point of interest identification on cluttered environments, feasibility analysis of the user command, the generation of the proper feedback to the user to solve potential ambiguous situations, and user experience and usability analysis of our system.
CitationSauras-Perez, P., Gil, A., Singh Gill, J., Pisu, P. et al., "VoGe: A Voice and Gesture System for Interacting with Autonomous Cars," SAE Technical Paper 2017-01-0068, 2017, https://doi.org/10.4271/2017-01-0068.
- “Continental Automotive – Advanced Driver Assistance Systems,” http://www.continental-automotive.com/www/automotive_de_en/themes/passenger_cars/chassis_safety/adas/. Last visited: October 2016.
- Juliussen, E. and Carlson, J., “Autonomous Cars – Not if, but when,” IHS Automotive, Jan. 2014.
- Nissan’s Autonomous Self Driving Car | Nissan USA. http://www.nissanusa.com/blog/autonomous-drive-car. Last visited: October 2016.
- Model S Autopilot Press Kit | Tesla Motors. https://www.teslamotors.com/presskit/autopilot. Last visited: October 2016.
- “Autonomous Cars – An industry influence study,” Appinions Inc., July 2014.
- “Self-Driving Cars: The Next Revolution,” KPMG, 2012.
- “Self-Driving Cars: Are We Ready?” KPMG, 2013.
- IEEE – News Releases, “Expert Members of IEEE Identify Driveless Cars as Most Viable Form of Intelligent Transportation, Dominating the Roadway by 2040 and Sparking Dramatic Changes In Vehicular Travel,” IEEE, Sept. 2012. http://www.ieee.org/about/news/2012/5september_2_2012.html. Last visited: October 2016.
- Montemerlo, M. et al., “Junior: The Stanford Entry in the Urban Challenge,” in Journal of Field Robotics, 2008.
- Google Self-Driving Car Project. https://www.google.com/selfdrivingcar/. Last visited: October 2016.
- IEEE – News Releases, “IEEE Survey Reveals Mass-Produced Cars Will Not Have Steering Wheels, Gas/Brakes Pedals, Horns, or Rearview Mirrors by 2035,” IEEE, July 2014. http://www.ieee.org/about/news/2014/14_july_2014.html. Last visited: October 2016.
- Lu, D. T. Z. et al., “Unlock and authentication for autonomous vehicles,” U.S. Patent 9,194,168, Nov. 24, 2015.
- Markoff, J., “Google’s Next Phase in Driverless Cars: No Steering Wheel or Brake Pedals,” The New York Times, May 2014. http://www.nytimes.com/2014/05/28/technology/googles-next-phase-in-driverless-cars-no-brakes-or-steering-wheel.html. Last visited: October 2016.
- McFarland, M., “Look inside Google’s new self-driving car,” The Washington Post, July 2015. https://www.washingtonpost.com/news/innovations/wp/2015/07/13/look-inside-googles-new-self-driving-car/. Last visited: October 2016.
- Dagmar, K. and Schmidt, A., “Design space for driver-based automotive user interfaces,” in Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 2009.
- Rodel, C. et al., “Towards Autonomous Cars: The Effect of Autonomy Levels on Acceptance and User Experience,” in Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 2014.
- De Ambroggi, L. and Kona, A., “Google's driverless car to boost revenues for semiconductor,” IHS Technology, June 2014. https://technology.ihs.com/502831/automotive-semiconductor-impression-220514. Last visited: October 2016.
- Schoettle, B. and Sivak, M., “A survey of public opinion about autonomous and self-driving vehicles in the U.S., the U.K., and Australia”, University of Michigan Transportation Research Institute UMTRI, July 2014.
- Rupp, J. D. and King, A. G., “Autonomous Driving – A Practical Roadmap,” SAE International, 2010.
- DeMattia, N., “Autonomous Driving: Coming in Small Doses,” BMW Blog, 2015, http://www.bmwblog.com/2015/04/07/autonomous-driving-coming-in-small-doses/. Last visited: October 2016.
- Davies, A., “Ford’s Skipping the Trickiest Thing About Self-Driving Cars,” WIRED, Nov. 2015. http://www.wired.com/2015/11/ford-self-driving-car-plan-google/. Last visited: October 2016.
- SAE On-Road Automated Vehicle Standards Committee. “Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems,” SAE International, 2014.
- Smith, B. W. and Svensson, J., “Automated and Autonomous Driving: Regulation under Uncertainty,” Organization for Economic Co-operation and Development (OECD), 2015.
- Smith, B. W., “Publications – NewlyPossible,” https://newlypossible.org/wiki/index.php?title=Publications. Last visited: October 2016.
- Casley, S. V. et al., “A study of public acceptance of autonomous cars,” Worcester Polytechnic Institute, Worcester, MA, USA, 2013.
- Fagnant, D. J. and Kockelman, K., “Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations for Capitalizing on Self-Driven Vehicles,” in Transportation Research Part A, July 2013.
- Kalra, N., Anderson, J. and Wachs, M., “Liability and Regulation of Autonomous Vehicle Technologies,” California PATH Program, Institute of Transportation Studies, University of California, Berkeley, April 2009.
- Jiang, T. et al., “Self_Driving Cars: Disruptive or Incremental?” in Applied Innovation Review, June 2015.
- California Department of Motor Vehicles, “Order to Adopt-Article 3.7 Autonomous vehicles,” August, 2014.
- Urmson, C., “The View from the Front Seat of the Google Self-Driving Car, Chapter 3,” https://medium.com/@chris_urmson/the-view-from-the-front-seat-of-the-google-self-driving-car-chapter-3-476ea9deed9a#.ny2z0sdhw. Last visited: October 2016.
- Hemmersbaugh, P. A., “Letter from NHTSA to Dr. Chris Urmson,” NHTSA, 2016. http://isearch.nhtsa.gov/files/Google%20--%20compiled%20response%20to%2012%20Nov%20%2015%20interp%20request%20--%204%20Feb%2016%20final.htm. Last visited: October 2016.
- “Federal Automated Vehicles Policy. Accelerating the Next Revolution in Road Safety.” USDOT, September 2016.
- Youngs, J., “2014 U.S. Automotive Emerging Technologies Study Results,” J.D. Power, McGraw Hill Financial, 2014. http://www.jdpower.com/cars/articles/jd-power-studies/2014-us-automotive-emerging-technologies-study-results. Last visited: October 2016.
- Adams, T., “Self-driving cars: from 2020 you will become a permanent backseat driver,” The Guardian, 2015. http://www.theguardian.com/technology/2015/sep/13/self-driving-cars-bmw-google-2020-driving. Last visited: October 2016.
- The Mercedes-Benz F 015 Luxury in Motion. https://www.mercedes-benz.com/en/mercedes-benz/innovation/research-vehicle-f-015-luxury-in-motion/. Last visited: October 2016.
- Rinspeed XchangeE, http://www.rinspeed.eu/concept-galery.php?cid=25. Last visited: October 2016.
- Sparkes, M., “Google forced to add steering wheel to driveless cars,” The Telegraph, Aug. 2014. http://www.telegraph.co.uk/technology/google/11051009/Google-forced-to-add-steering-wheel-to-driverless-cars.html. Last visited: October 2016.
- Bolt, R. A., “Put that there: Voice and Gesture at the Graphics Interface,” in Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, ACM, 1980.
- Kranstedt, A. et al., “Deixis: How to determine demonstrated objects using a pointing cone,” in Gesture in Human-Computer Interaction and Simulation. Springer Berlin Heidelberg, 2005, pp. 300-311.
- Holzapfel, H., Nickel, K. and Stiefelhagen, R., “Implementation and Evaluation of a Constraint-Based Multimodal Fusion System for Speech and 3D Pointing Gestures." In Proceedings of the 6th International Conference on Multimodal Interfaces, ACM, 2004.
- Kowadlo, G., Ye, P., and Zukerman, I., “Influence of Gestural Salience on the Interpretation of Spoken Requests,” in INTERSPEECH. 2010.
- Muller, C. and Weinberg, G., “Multimodal input in the car, today and tomorrow” in IEEE MultiMedia, IEEE, 2011.
- Pfleging, B., Schneegass, S. and Schmidt, A., “Multimodal Interaction in the Car: Combining Speech and Gestures on the Steering Wheel,” in Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2012.
- Hobbs, N. et al., “Gesture-based automotive controls,” U.S. Patent 8,942,881, Jan. 27, 2015.
- King, A. G., “Interacting with vehicle controls through gesture recognition,” U.S. Patent Application 13/366,388, Aug. 8, 2013.
- Riener, A. et al., “Standardization of the In-Car Gesture Interaction Space,” in Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2013.
- Vaghefinazari, P. et al., “System and Method for incorporating Gesture and Voice Recognition into a Single System,” U.S. Patent US9211854 B2, Dec. 15, 2015.
- Rumelin, S., Marouane, C., Butz, A., “Free-hand pointing for identification and interaction with distant objects,” in Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2013.
- Matsumura, K. et al., “Stick'N Conversation: Stick In-car Conversation into Places Using Multi Person Finger Pointing Gestures,” in Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2015.
- McCall, C. E. et al., “Gesture actuated point of interest information systems and methods,” U.S. Patent Application 12/430,389, Oct. 28, 2010.
- Traore, I. ed. “Continuous Authentication Using Biometrics: Data, Models, and Metrics,” IGI Global, 2011.
- Kinect – Windows app development. https://dev.windows.com/en-us/kinect. Last visited: October 2016.
- EmguCV: OpenCV in .NET (C#, VB, C++ and more). http://www.emgu.com/. Last visited: October 2016.
- OpenCV. http://opencv.org/. Last visited: October 2016.
- Arduino – Home. https://www.arduino.cc/. Last visited: October 2016.
- Minoru 3D. http://www.minoru3d.com/. Last visited: October 2016.
- Bradski, G. and Kaehler, A., “Learning OpenCV: Computer vision with the OpenCV library,” O'Reilly Media, Inc., 2008.
- Jiang, Y. et al., “Target object identification and localization in mobile manipulations,” in 2011 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 2011.
- Wang, M. and Latotzky, D., “Driving an autonomous car with eye tracking,” Free University of Berlin, 2010.
- Göhring, D. et al., “Semi-autonomous car control using brain computer interfaces,” in Intelligent Autonomous Systems 12, Springer Berlin Heidelberg, 2013, pp. 393-408.
- Richarz, J. et al., “There you go!-estimating pointing gestures in monocular images for mobile robot instruction,” in the 15th IEEE International Symposium on Robot and Human Interactive Communication, IEEE, 2006.
- Droeschel, D. et al., “Learning to interpret pointing gestures with a time-of-flight camera,” in Proceedings of the 6th international conference on Human-robot interaction, ACM, 2011.
- Kang, S. et al., “Do you see what I see: towards a gaze-based surroundings query processing system,” in Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2015.
- Kim, Y. H. and Misu, T., “Identification of the driver's interest point using a head pose trajectory for situated dialog systems,” in Proceedings of the 16th International Conference on Multimodal Interaction, ACM, 2014.
- Misu, T. et al., “Situated language understanding for a spoken dialog system within vehicles,” in Computer Speech & Language, 34(1), 2015, pp. 186-200.
- Grossman, T. and Balakrishnan, R., “The design and evaluation of selection techniques for 3D volumetric displays,” in Proceedings of the 19th annual ACM Symposium on User Interface Software and Technology, ACM, 2006.
- Steed, A., “Towards a general model for selection in virtual environments,” in IEEE Symposium on 3D User Interfaces, IEEE, 2006.
- Zobl, M. et al., “A usability study on hand gesture controlled operation of in-car devices,” in Proceedings, HCI, 2001.
- Tchankue, P. et al., “Design and evaluation of a multimodal interface for in-car communication systems,” in Proceedings of the 2010 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, ACM, 2010.
- Baron, A. and Green, P., “Safety and usability of speech interfaces for in-vehicle tasks while driving: A brief literature review,” No. UMTRI-2006-5. University of Michigan, Transportation Research Institute, 2006.
- Hua, Z. and Wei, L. Ng., “Speech recognition interface design for in-vehicle system,” in Proceedings of the 2nd International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2010.
- Alvarez, I. et al. “Designing driver-centric natural voice user interfaces,” in Adjunct Proceedings of AutomotiveUI, ACM, 2011.
- Pickering, C. et al. “A review of automotive human machine interface technologies and techniques to reduce driver distraction,” in 2nd International Conference on Institution of Engineering and Technology (IET), 2007.
- Riener, A. and Rossbory, M., “Natural and Intuitive Hand Gestures: A Substitute for Traditional Vehicle Control?” in Proceedings of the AutomotiveUI, ACM, 2011.
- Lee, J. C., “In search of a natural gesture,” in ACM Crossroads, 16.4 (2010): 9-12.
- Malizia, A. and Bellucci, A., “The artificiality of natural user interfaces,” in Communications of the ACM 55, no. 3 (2012): 36-38.
- Norman, D. A., “Natural user interfaces are not natural,” in Interactions of the ACM 17, no. 3 (2010): 6-10.
- Norman, D. A. and Nielsen, J., “Gestural interfaces: a step backward in usability,” in Interactions of the ACM 17, no. 5 (2010): 46-49.
- Sauras-Perez, P. et al., “Variability analysis of in-car gesture interaction,” in 2014 International Conference on Connected Vehicles and Expo (ICCVE). IEEE, 2014.
- Głomb, P. et al., “Choosing and modeling the hand gesture database for a natural user interface,” in Gesture and sign language in human-computer interaction and embodied communication, Springer Berlin Heidelberg, 2011.
- Grandhi, S. A. et al., “Understanding naturalness and intuitiveness in gesture production: insights for touchless gestural interfaces,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2011.