Abstract: The finite element model of a non-associated gear compressor rotor system was established with considering the influence of the gears engagement. To study the characteristics of the gear-rotor system, the single axis of the whole system are successively analyzed. Dynamic characteristics of the whole machine with engaging gears are tested through bending, torsional and coupled vibrations. The results show that new system frequency arises due to the existence of the gear meshing effect; the torsional effect of sleeve on the coupled system is mainly reflected in the third and the fourth order; the coupled effect of gears is relatively weak in the coupled characteristics of bend and torsion. To avoid design defects of the system, the gear meshing effect must be fully considered. This work provids a theoretical reference for fault diagnoses, vibration control and stability design of the rotor-bearing system.