The sweeper vehicle plays a very key role in maintaining the urban environment. If the sweeper vehicle can accurately and efficiently identify and classify the ground garbage in the working process, it can greatly improve the working efficiency of the sweeper vehicle and reduce the consumption of manpower. Although the deep learning algorithm based on DUC and PSPNet has high accuracy, the recognition speed is low. ENet is a lightweight network, which greatly improves efficiency, but significantly sacrifices accuracy. This paper presents an improved real-time detection lightweight network based on PSPNet, which takes into account the operation speed and accuracy. The network takes PSPNet as the backbone network, and increases the stride in the convolution process, to reduce the size of the feature map and reduce the amount of calculation. The Elu is selected as the activation function of neurons to select features and suppress noise, which improves the nonlinear fitting ability of the network. The parameters that affect the computational complexity of the convolution layer are adjusted to achieve a balance between detection rate and accuracy. By using a self-made dataset, the method of transfer learning is used for training. Experimental results show that the improved resnet50-based PSPNet algorithm has a higher mean pixel accuracy than the PSPNet algorithm. Moreover, the reasoning time of this algorithm is 23ms, which is 8.7% faster than the PSPNet algorithm. They also show that the improved MobileNet-based PSPNet algorithm has a higher MIoU than the PSPNet algorithm and that the reasoning time of this algorithm is 9ms, which is 22.2% faster than the PSPNet algorithm. The network ensures the balance between recognition accuracy and real-time detection rate, which can effectively help the sweeper vehicle to classify garbage automatically and can provide a reference for improving the semantic segmentation algorithm of real-time detection of intelligent vehicles.