Drivers often interact with partial automation (SAE Level 2) systems, initiating transfer of control (TOC) either by handing control over to the automation or by taking it back. Accurately predicting these interactions may inform the design of future automation systems that adapt proactively to the operating context, enhance comfort, and ultimately may improve safety. We present a context-aware framework that generates a unified driver–vehicle–environment representation by fusing data from in-cabin video of the driver and of the forward roadway with vehicle kinematics, driver glance, and hands-on-wheel behaviors. This representation was encoded in a hierarchical Graph Neural Network that classified driver-initiated TOCs to: (i) Manual-to-automation and (ii) Automation-to-manual transitions and predicted time-to-TOC. Shapley-based explainable AI was used to quantify how the importance of behavioral, contextual, and kinematic cues evolved in the seconds preceding a TOC. Analysis of a naturalistic dataset of 1,565 driver-initiated TOCs from 16 experienced drivers revealed distinct patterns. Manual-to-automation transitions were preceded by lane count increases, acceleration, and spikes in glances to the instrument-cluster. In contrast, Automation-to-manual transitions were associated with lane count reductions, higher surrounding-vehicle density, deceleration, reduction in secondary-task engagement, and higher steering wheel control. Together, these patterns highlight key cues for predicting the TOC type and time-to-TOC. Using environment-only features, the classifier achieved 78% accuracy; adding vehicle kinematics increased accuracy to 84%, and incorporating driver behavior features further improved prediction to 90%. Across prediction horizons, the Manual-to-automation TOC was consistently predicted more accurately than the automation-to-manual TOC. Shapley analyses underscore that driver behavior provided the strongest cues for predicting TOCs, highlighting the value of fusing driving context with information obtained from monitoring the driver behavior to anticipate the type of driver-automation interaction and its timing.