Transcritical Rankine cycle (TRC) is a promising technology for the engine waste heat recovery due to its good temperature matching ability for the waste heat sources. As for the high-temperature engine exhaust, working fluids selection has been an essential issue without a good solution. It was found in this research that mixtures of CO2 and small molecule hydrocarbons are the potential working fluids for the engine waste heat recovery, since they have good chemical stability and thermal performance. Besides, CO2 can be used as the retardant to suppress the flammability of hydrocarbons to ensure safety. In this research, CO2 mixed with five small molecule hydrocarbons are proposed as the working fluids. A thermodynamic model of TRC system is established to evaluate the thermal performance of those mixtures. The effects of mass fraction of CO2, turbine inlet temperature and pressure are investigated. The influence of composition shift is also discussed. The results show that, 65% CO2 in the mixtures is a tremendous improvement for the system safety, comparing to pure hydrocarbons. Compared with the pure CO2 TRC system, the thermal performance of the system can be effectively improved by mixing appropriate hydrocarbons with CO2, and the operation pressure can be decreased, which is beneficial for the future application. Take CO2/n-Pentane for example, the thermal efficiency and net power output can reach up to 16.4% and 16.06 kW, increasing by 34.4% and 13.3% compared with pure CO2. And the composition shift of those mixtures has little effect on the performance of TRC, but can improve the TRC system safety.