A key issue regarding feature extraction is the capability of a technique to extract distinctive features to represent facial expressions while requiring a low computational complexity. In this study, the authors propose a novel approach for appearance-based facial feature extraction to perform the task of facial expression recognition on video sequences. The proposed spatiotemporal texture map (STTM) is capable of capturing subtle spatial and temporal variations of facial expressions with low computational complexity. First, face is detected using Viola–Jones face detector and frames are cropped to remove unnecessary background. Facial features are then modelled with the proposed STTM, which uses the spatiotemporal information extracted from three-dimensional Harris corner function. A block-based method is adopted to extract the dynamic features and represent the features in the form of histograms. The features are then classified into classes of emotion by the support vector machine classifier. The experimental results demonstrate that the proposed approach shows superior performance compared with the state-of-the-art approaches with an average recognition rate of 95.37, 98.56, and 84.52% on datasets containing posed expressions, spontaneous micro-expressions, and close-to-real-world expressions, respectively. They also show that the proposed algorithm requires low computational cost.