Abstract: Reliability and reliability based sensitivity problem of a gear coupling vibration system was solved with the reliability model established on the basis of the dynamical model of the gear system. Combining the nonlinear dynamical response with the reliability analysis technique, the fourth moment technique and Edgeworth series method are used to calculate the reliability of the vibrating amplitudes and the transmission error. The system reliability is obtained with independence of failure modes. The reliability based sensitivities with respect to the mean and variance of the random parameters are analyzed based on the study of the system reliability. The as-proposed method is effective for solving the reliability problems of gear vibration systems and governing the structures optimized design, which is useful to reduce the instability issue caused by vibration.