Vehicle detection has been a fundamental problem in the research of Intelligent Traffic System (ITS), especially in urban driving environment. Over the past decades, vision-based vehicle detection has got a considerable attention. In addition to the rich appearance information, the stereo vision method also provides depth information, which could achieve higher accuracy and precision. In this paper, a hybrid method for stereo vision-based real-time vehicle detection in urban environment is proposed. Firstly, we extract vehicle features and generate the Region of Interest (ROI). Semi-global Matching (SGM) algorithm is then utilized on the ROIs to generate disparity maps and get the depth information, which could be used to compute the width of each ROI. The noise regions, always with obvious depth variation in the disparity maps are excluded by the clustering approach. Finally, we use Histogram of Oriented Gradient (HOG) feature and Support Vector Machine (SVM) to verify the final vehicles in the candidate sets. To optimize the system further, we lead in the multi-scale classifier with the detection accuracy increasing dramatically. Experimental results show that the proposed method could achieve more than 20 fps in urban environment and the system could effectively remove the interference caused by trees, guardrails and buildings, which demonstrate its good application prospect. Further, our method can be applied to ADAS on the collision warning system and active braking system for front obstacle detection.