Face recognition technique has obtained great progress and excellent results on public data sets. However, traditional algorithms suffer from various changes such as illumination, expression, and misalignment in practical applications. To solve these problems, this study proposes a novel face recognition algorithm simultaneously resolves these challenges. The key idea is reducing the influence of illumination and expression through the aligning procedure. As a result, illumination, expression, and misalignment can be greatly ignored in the recognition procedure. The contributions of this study are two folds. (i) The construction of the shape constrained illumination pattern (SCIP), which models the illumination variation with robustness to expression change. (ii) SCIP-based face recognition algorithm which can deal with illumination, expression, and image misalignment simultaneously. Systematic evaluations conducted on public databases demonstrate that the proposed algorithm is robust to illumination, expression, and misalignment with better performance than state-of-the-art algorithms.