Abstract: Incremental harmonic balance (IHB) method could not solve the coupled bending and torsional vibration equation of positive semidefinite in planetary gear transmission mechanism directly， and microrelative displacement was applied to describe relative motion between the parts in the planetary gear system. The motion transfer matrix was used to transform positive semidefinite equation into positive definite equation. Meanwhile， considering the timevarying nonlinear characteristics of gear mesh stiffness， the IHB method was used to analyze the nonlinear dynamics response in the planetary gear system. The correctness and efficiency of the proposed method were verified by comparison with Newmarkβ method.

Key words: planetary gear, bendingtorsional vibration, incremental harmonic balance method, nonlinear, dynamic response