A novel approach for face detection in video is proposed. Vision Similarity is defined to measure the relationship between video frames. Then the candidate face regions are obtained in previous video frame, which has similarity with the current frame. The local Successive Mean Quantization Transform (SMQT) and Sparse Network of Winnows (SNoW) are used on candidate regions, and the skin color segmentation is implemented for non-vision similar frames. Experimental results show that, with the candidate region, the detection algorithm has better performance on real-time procession than the previous one. And the detection accuracy is also higher than it as well.