Drivers often interact with partial automation (SAE Level 2) systems, initiating transfer of control (TOC) either by handing control over to the automation or by taking it back. Accurately predicting these interactions may inform the design of future automation that adapts proactively to the operating context, enhances comfort, and improves safety. We present a context-aware framework that generates a unified driver–vehicle–environment representation by fusing data from videos of the in-cabin and forward road with vehicle kinematics, and driver behaviors (annotations of driver glance, hands-on-wheel, and secondary-task engagement). This representation is encoded in a hierarchical Graph Neural Network that classifies driver-initiated TOC to: (i) Manual-to-Automation (M2A) and (ii) Automation-to-Manual (A2M) and predicts time-to-TOC. Shapley-based explainable AI was used to quantify how the importance of behavioral, contextual, and kinematic cues evolved in the seconds preceding a TOC. Analysis of a naturalistic dataset of 1,565 driver-initiated TOCs from 16 experienced drivers revealed distinct patterns. M2A transitions were preceded by lane count increases, vehicle acceleration, and spikes in instrument-cluster glances. In contrast, A2M transitions were associated with lane reductions, higher surrounding-vehicle density, vehicle deceleration, reduced secondary-task engagement, and higher hands-on-wheel levels. Together, these patterns highlight key cues for predicting TOC type and time-to-TOC. Using environment-only features, the classifier achieved 78% accuracy; adding vehicle kinematics increased accuracy to 86%, and incorporating driver behavior features further improved prediction to 92%. Across prediction horizons, M2A was consistently predicted more accurately than A2M. Shapley analyses underscore that driver behavior provided the strongest cues for predicting TOCs. This highlights the value of fusing driving context and driving kinematics with information obtained from monitoring driver behavior to anticipate the type and timing of driver-automation interaction.