Abstract: Hand is a highly variable organ and hand features are easily affected by environmental factors. Considering the characteristics of hand gesture, a novel hand gesture recognition algorithm based on Fourier descriptor and BP neural network is presented. According to the complementarity of the YCbCr and Nrg color models and the background model that is available to exclude the similar skin colors from the complicated background, the hand gesture region is extracted by the segmentation of hand gestures through multi-feature integration. Then, the advantages of the Fourier descriptor which can describe well the profiles and of the BP neural network which is strong at self-learning are taken and combined together to recognize hand gestures. The experimental results showed that the method proposed has better robustness and higher recognition rate.