To address the measurement challenges posed by the large diameters and wide spans of engine crankshaft holes, cylinder holes, and camshaft holes, as well as the abnormal laser measurement data caused by oil film adhesion and local protrusion structures on their inner walls, this paper proposes a non-contact coaxiality measurement method based on a laser displacement sensor (LDS) and a threshold-modified least squares ellipse fitting (TMLSE) algorithm. The method first preprocesses the data through median filtering and then employs an adaptive distance threshold based on maximum inter-class variance to eliminate outliers caused by oil film scattering and protrusions, achieving robust fitting of the cross-sectional centers. Subsequently, the datum axis is established using the least squares midline method, ultimately enabling the evaluation of coaxiality error. In the simulation experiments, a dual-hole model with an inner diameter of 100 mm and a spacing of 700 mm was simulated, where the right hole was translated by 1.5 mm to introduce coaxiality error. Each cross-sectional point cloud included 10% protrusion points (protrusion height of 10 mm) along with random noise and data loss simulating the effects of oil film. To assess the robustness of the method, a systematic analysis was conducted on the influence of laser incidence angles [80°,89°]on measurement accuracy. The results show that across different incidence angles, the maximum deviation between the coaxiality error obtained by the TMLSE method and the theoretical value is only 0.1582 mm, which is significantly better than that of the RANSAC, MZC, and MIC methods. Meanwhile, TMLSE demonstrates stable and efficient computational speed. The simulation verifies that this method offers higher fitting accuracy and robustness under both optical interference and variations in installation angle, making it suitable for the precision measurement of key engine hole systems.