The driver's face detection and tracking method important for Advanced Driver Assistance Systems (ADAS) and autonomous driving in various situations. The deep SORT algorithm has integrated appearance information, the motion model and the intersection-over-union (IOU) distance methods, and has been applied to face tracking, but it depends on detection information in every frame. Once the detection information lacks, the deep SORT algorithm will wait until the target detects bounding boxes appear again, even if the target didn’t disappear or shield. Hence, we propose to use a new tracker that not completely depend on the detection algorithm to cascade with the deep SORT algorithm to realize stable driver's face tracking. At first, the driver's face detection and tracking will be accomplished by the MTCNN-deep-SORT algorithm. Multi-task convolutional neural network (MTCNN) will complete the driver's face detection, and detected face bounding boxes will be transferred into deep SORT tracking algorithm, at this step, we will get the driver's face detection and tracking bounding boxes. Subsequently, the detection of bounding boxes is transferred to the kernel correlation filtering (KCF) tracking algorithm. When the driver's face is not detected, the KCF tracker will use the last frame detection information to track the driver's face at the current frame, transfer the tracking information into the deep SORT algorithm and reidentify the driver's face in the current frame. In case that the detection is not lost, we only record detection information in the KCF tracker and do not start the tracker. So, we have successfully cascaded the deep learning tracking method and the traditional feature tracking method. Volunteer experiments on 20 drivers show that our cascaded tracking algorithm method achieves more stable driver's face detection and tracking when the driver's head posture changes greatly.