Abstract: To detect a figure accurately in an image with great background noise, an improved Hough transform with parameters screened by least mean square error (LMSE) method is proposed as follows. First, Hough transform is introduced to obtain the total number of votes in parametric space, and the parameters of which the vote is greater than the threshold value for voting are all screened out preliminarily. Then, the mean square errors of image edges are separately calculated corresponding to these parameters, and a set of parameters with LMSE is chosen as the final result of such detection. The approach proposed is compared with the Hough transform with fetching the maximum votes by applying them separately to the localization of the outer iris limbus. Experimental results showed that the approach proposed can offer more reasonable findings with higher resistance to noise and wider range of applications.