Gearbox fault diagnosis is one of the core research areas in the field of rotating machinery condition monitoring. The signal processing-based bearing fault diagnosis in the gearbox is considered as challenging as the vibration signals collected from acceleration transducers are, in general, a mixture of signals originating from an unknown number of sources, i.e. an underdetermined blind source separation (UBSS) problem. In this study, an effective UBSS-based algorithm solution, that combines empirical mode decomposition (EMD) and kernel independent component analysis (KICA) method, is proposed to address the technical challenge. Firstly, the nonlinear mixture signals are decomposed into a set of intrinsic mode function components (IMFs) by the EMD method, which can be combined with the original observed signals to reconstruct new observed signals. Thus, the original problem can be effectively transformed into over-determined BSS problem. Then, the whitening process is carried out to convert the over-determined BSS into determined BSS, which can be solved by the KICA method. Finally, the ant lion optimization (ALO) is adopted to further enhance the performance of the EMD-KICA method. The proposed solution is assessed through simulation experiments for non-linearly mixed bearing vibration signals, and the numerical result demonstrates the effectiveness of the proposed algorithmic solution.