To improve the safety of automated driving, the paramount target of this intelligent system is to detect and segment the obstacle such as car and pedestrian, precisely. Object detection in self-driving vehicle has chiefly accomplished by making decision and detecting objects through each frame of video. However, there are diverse group of methods in both machine learning and machine vision to improve the performance of system. It is significant to factor in the function of the time in the detection phase. In other word, considering the inputs of system, which have been emitted from eclectic type of sensors such as camera, radar, and LIDAR, as time-varying signals, can be helpful to engross ‘time’ as a fundamental feature in modeling for forecasting the object, while car is moving on the way. In this paper, we focus on eliciting a model through the time to increase the accuracy of object detection in self-driving vehicles. In fact, we designed a deep recurrent neural network, which is fed by the output of a deep convolutional neural network. Eventually, the proposed system could make a decision about each front vehicle not only by utilizing the data during a predefined span of the video, but also enhance the performance of system to determine the object as whether a car or not. Results shows that the performance of system compared to convolutional neural network based vehicle detector, is more precise in the case of discerning cars.