An Inline 4-cylinder engine is equipped with second-order balance shafts. When the engine is running under no-load acceleration conditions, the gear system of the balance shaft generated whine noise. In this paper, an analysis and experiment method for reducing the whine noise is presented. First, a flexible multi-body dynamic model of the engine is established, which includes shaft and casing deformation, micro-modification of the gears. Taking the measured cylinder pressure as input, the load on each gear of balance shaft gear system is calculated. In addition, the influence of tooth surface micro-modification on the meshed noise was analyzed. The results show that the dynamic meshing force between the crank gear and the shim gear is large under the original tooth surface micro-modification parameters, which is the main reason of the whine noise. The torsional vibration at the crankshaft nose and vibration acceleration at the cylinder block was measured during no-load conditions, and the measured results were compared with the calculated results, which validate the established model. Secondly, the mesh misalignment and load calculated in the model are used as inputs to optimize the tooth surface micro-modification parameters. Taking minimizing the peak-to-peak of loaded transmission error and the maximum contact stress on the tooth surface as optimization objectives, the micro-modification parameters as optimization variables, the NSGA-II (Non-dominated Sorting Genetic Algorithm II) algorithm was used to conduct multi-objective optimization. A set of tooth surface micro-modification parameters was optimized, and the peak-to-peak loaded transmission error is reduced by 80%. According to the optimized parameters, the new scissor gears was fabricated and installed on the balance shaft. The experiment results showed that the whine evaluation index TNR (Tone-to-Noise Ratio) is reduced from 2.2dB to -1.4dB, which solved the whine noise.