The track irregularity has an important influence on the comfort, noise, and safety of the vehicle. At present, track unevenness is still measured using a track inspection trolley and other equipment, which is less efficient. In order to measure the track roughness on the existing operation lines, an evaluation method of track roughness based on inertial reference is proposed, and the method of calculating track roughness through the acceleration signal of the existing vehicle axle box is studied. The track roughness is obtained by integrating the acceleration signal twice in the time and frequency domains, respectively, and the simulation signal is established for theoretical verification, which is verified by experiments using the acceleration signal of an intelligent vehicle axle box. The analog signal is used for theoretical verification, and the axle box acceleration signal of the intelligent locomotive and rolling stock is used for experimental verification. The experimental results show that the axle-box acceleration signal of the rail vehicle can accurately reflect the characteristics of track unevenness, and at the same time, after removing the zero drift of the sensor and the trend term generated in the integration process, the calculation results of the two methods are highly consistent, which is of great significance for the state assessment of the rail line.