Abstract: In order to improve the DOA estimation accuracy of mixed signals and reduce array aperture loss， a high-precision DOA estimation algorithm based on oblique projection operator is proposed. The proposed algorithm estimates the independent and coherent signals of the mixed signals in two stages. Firstly， the covariance matrix of the data received by the array element is processed by the estimating signal parameter via rotational invariance techniques(ESPRIT)， and the DOA estimates of the independent signals in the mixed signals are calculated. Then， the algorithm uses the oblique projection operator to remove the independent signal information in the mixed signals to obtain a new covariance matrix. The signal subspace of the newly obtained covariance matrix is used for decoherence processing. Finally， the ESPRIT algorithm is used to calculate the DOA estimates of the coherent signals. Simulation results show that the proposed algorithm has higher accuracy than traditional mixed signal DOA estimation algorithms in the case of low signal-to-noise ratio and small signal incidence interval， and effectively reduces array aperture loss. Under the condition of different sampling snapshot numbers， the proposed algorithm also shows stronger robustness.

Key words: mixed signals; DOA(direction of arrival) estimation; oblique projection operator; signal subspace; ESPRIT algorithm