As intelligent automated vehicle technologies evolve, there is a greater need to understand and define the role of the human user, whether completely hands-off (L5) or partly hands-on. At all levels of automation, the human occupant may feel anxious or ill-at-ease. This may reflect as higher stress/workload. The study in this paper further refines how perceived workload may be determined based on occupant physiological measures. Because of great variation in individual personalities, age, driving experiences, gender, etc., a generic model applicable to all could not be developed. Rather, individual workload models that used physiological and vehicle measures were developed. Unlike some existing methods of workload estimation where one, or a few signals are used, such as electroencephalography (EEG), electrocardiography (ECG), we developed intelligent systems that use multiple physiological and vehicle signals based on an end-to-end deep neural learning architecture to make a robust estimation of workload. The deep neural learning system, MTS-CNN, is designed to learn workload patterns from synchronized, heterogeneous temporal signals. All data collected for training and testing are from real-world driving trips along the same route which comprised urban local roads and highways. Data from twenty participants whose driving experience ranged from a few months to several years were collected and analyzed. The experimental results indicate that the proposed driver workload estimation model is capable of learning well from the combined temporal physiological and vehicle signals and good performance was obtained on workload estimation.