This content is not included in your SAE MOBILUS subscription, or you are not logged in.
Development of an Autonomous Vehicle Control Strategy Using a Single Camera and Deep Neural Networks
ISSN: 0148-7191, e-ISSN: 2688-3627
Published April 03, 2018 by SAE International in United States
Annotation ability available
Autonomous vehicle development has benefited from sanctioned competitions dating back to the original 2004 DARPA Grand Challenge. Since these competitions, fully autonomous vehicles have become much closer to significant real-world use with the majority of research focused on reliability, safety and cost reduction. Our research details the recent challenges experienced at the 2017 Self Racing Cars event where a team of international Udacity students worked together over a 6 week period, from team selection to race day. The team’s goal was to provide real-time vehicle control of steering, braking, and throttle through an end-to-end deep neural network. Multiple architectures were tested and used including convolutional neural networks (CNN) and recurrent neural networks (RNN). We began our work by modifying a Udacity driving simulator to collect data and develop training models which we implemented and trained on a laptop GPU. Then, in the two days between car delivery and the start of the competition, a customized neural network using Keras and Tensorflow was developed. The deep learning network algorithm predicted car steering angles using a single front-facing camera. Training and deployment on the vehicle was completed using two GTX 1070s since a cloud GPU computing instance was neither available nor feasible. Using the proposed methods and working within the competition’s strict requirements, we completed several semi-autonomous laps and the team remained competitive. The results of the competition indicated that autonomous vehicle command and control can be achieved in a limited form using a single-camera with a short engineering development timeline. This approach lacks robustness and reliability and therefore, a semantic segmentation network was developed using feature extraction from the YOLOv2 network and the CamVid dataset with a correction for the unbalanced occurrence of the different classes. Currently 31 classes can be reliably detected and classified allowing for a more complex and robust decision making architecture.
CitationNavarro, A., Joerdening, J., Khalil, R., Brown, A. et al., "Development of an Autonomous Vehicle Control Strategy Using a Single Camera and Deep Neural Networks," SAE Technical Paper 2018-01-0035, 2018, https://doi.org/10.4271/2018-01-0035.
- Charette, R.N., “This Car Runs on Code,” IEEE Spectrum 46:3, 2009.
- Bishop, R., “Intelligent Vehicle Technology and Trends,” trid.trb.org, 2005.
- Gusikhin, O.,Filev, D., andRychtyckyj, N., “Intelligent Vehicle Systems: Applications and New Trends,” . In: Informatics in Control Automation and Robotics. (Berlin Heidelberg, Springer, 2008), 3-14.
- National Highway Traffic Safety Administration, “National Motor Vehicle Crash Causation Survey,” (U.S. Department of Transportation, 2008).
- National Highway Traffic Safety Administration, “2016 Fatal Motor Vehicle Crashes: Overview,” (U.S. Department of Transportation, 2017).
- U S House of Representatives, “MAP-21 Conference Report to accompany H.R. 4348,” 2012.
- Bengler, K.,Dietmayer, K.,Farber, B.,Maurer, M. et al., “Three Decades of Driver Assistance Systems: Review and Future Perspectives,” IEEE Intelligent Transportation Systems Magazine 6:6-22, 2014.
- Mahajan, H.S.,Bradley, T., andPasricha, S., “Application of Systems Theoretic Process Analysis to a Lane Keeping Assist System,” Reliability Engineering and System Safety 167:177-183, 2017.
- Kukkala, V.K.,Tunnell, J.,Pasricha, S.,Bradley, T., “A Survey of Advanced Driver Assistance Systems and Current Challenges. In Review”.
- Fagnant, D.J. andKockelman, K., “Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations,” Transportation Research. Part A, Policy and Practice 77:167-181, 2015.
- Thrun, S.,Montemerlo, M.,Dahlkamp, H.,Stavens, D. et al., “Stanley: The Robot that Won the DARPA Grand Challenge,” Journal of Field Robotics 23:661-692, 2006.
- Urmson, C.,Andrew Bagnell, J.,Baker, C.R.,Hebert, M. et al., “Tartan Racing: A Multi-Modal Approach to the DARPA Urban Challenge,” 2007.
- Cameron, O., “Race Self-Driving Cars With Udacity! - Udacity Inc - Medium,” https://medium.com/udacity/race-self-driving-cars-with-udacity-42ae12e545c1.
- Gundling, C., “Our Very Own Grand Challenge - Udacity Inc - Medium,” https://medium.com/udacity/our-very-own-grand-challenge-b004a9863024.
- Sun, Z.,Bebis, G., andMiller, R., “On-Road Vehicle Detection: A Review,” IEEE Transactions on Pattern Analysis and Machine Intelligence 28:694-711, 2006.
- Rosenblatt, F., “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain,” Psychological Review 65:386-408, 1958.
- Hopfield, J.J., “Neural Networks and Physical Systems with Emergent Collective Computational Abilities,” Proceedings of the National Academy of Sciences of the United States of America 79:2554-2558, 1982.
- Rumelhart, D.E.,McClelland, J.L., and PDP Research Group, editors, “Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Vol. 1: Foundations,” (Cambridge, MIT Press, 1986).
- Demuth, H.B.,Beale, M.H.,De Jess, O., andHagan, M.T., “Neural Network Design,” (USA, Martin Hagan, 2014).
- Sallab, A.E.L.,Abdou, M.,Perot, E., andYogamani, S., “Deep Reinforcement Learning Framework for Autonomous Driving,” Electronic Imaging 2017:70-76, 2017.
- Ohn-Bar, E. andTrivedi, M.M., “Looking at Humans in the Age of Self-Driving and Highly Automated Vehicles,” IEEE Transactions on Intelligent Vehicles 1:90-104, 2016.
- “Self Driving Car Engineer Nanodegree,” https://www.udacity.com/course/self-driving-car-engineer-nanodegree--nd013.
- Bojarski, M.Del Testa, D., ,Dworakowski, D.,Firner, B. et al., “End to End Learning for Self-Driving Cars,” http://arxiv.org/abs/1604.07316, 2016.
- Udacity, “Udacity’s Self-Driving Car Simulator,” https://github.com/udacity/self-driving-car-sim.
- PolySync Core, https://polysync.io/core/.
- LeCun, Y., Others, “LeNet-5, Convolutional Neural Networks,” http://yann. lecun. com/exdb/lenet, 2015.
- Krizhevsky, A.,Sutskever, I., andHinton, G.E., “ImageNet Classification with Deep Convolutional Neural Networks,” . In: Pereira F.,Burges C.J.C.,Bottou L., andWeinberger K.Q., editors. Advances in Neural Information Processing Systems 25. (Curran Associates, Inc., 2012), 1097-1105.
- PolySync. “Open Source Car Control,” https://github.com/PolySync/OSCC/wiki.
- PolySync, “Hardware,” https://github.com/PolySync/oscc/wiki/Hardware-Main.
- Redmon, J.Farhadi, A. and, “YOLO9000: Better, Faster, Stronger,” arXiv preprint arXiv:1612. 08242, 2016.
- Brostow, G.J.,Fauqueur, J., andCipolla, R., “Semantic Object Classes in Video: A High-Definition Ground Truth Database,” Pattern Recognition Letters 30:88-97, 2009.
- Brostow, G.J.,Shotton, J.,Fauqueur, J., andCipolla, R., “Segmentation and Recognition Using Structure from Motion Point Clouds,” . In: Computer Vision - ECCV 2008. (Berlin, Heidelberg, Springer, 2008), 44-57.