An analysis of drivers' facial expressions and subsidiary behavior events (e.g., yawning, self-touching hand motions, etc.) revealed a significant correlation between the struggle against sleepiness and the frequency of occurrence of such events. We counted drivers' subsidiary behavior events by video analysis and defined nine categories of events related to the mouth, hands, head, shoulders, body and eyes. Mouth-related events were further categorized as yawning, stifling a yawn, exhaling and deep breathing. Yawning and self-touching hand motions in particular were observed in relatively large numbers among subsidiary behavior events. Based on this observation, we created an algorithm for detecting yawning and self-touching hand motions using a monocular camera and calculated the frequency of these subsidiary behavior events. In experiments, we compared the frequency of the subsidiary behavior events at the outset of driving and after the passage of time. As a result, we found time frames with a higher frequency of subsidiary events than at the outset, indicating an imminent decline in alertness based on a comparison with drivers' facial expressions. Our results show that the proposed method can detect a decrease in alertness earlier than the conventional eye closure rate method.