Convolutional neural networks (CNNs)-based deep features have been demonstrated with remarkable performance in various vision tasks, such as image classification and face verification. Compared with the hand-crafted descriptors, deep features exhibit more powerful representation ability. Typically, higher layer features contain more semantic information, while lower layer features can provide more low-level description. In addition, it turns out that the fusion of different layer features will lead to superior performance. Here, we propose a novel approach for human ear identification by combining hierarchical deep features. First, hierarchical deep features are extracted from ear images using CNN pre-trained on large-scale data set. To enhance the feature representation and reduce the high dimension of deep features, the discriminant correlation analysis (DCA) is adopted for fusing deep features from different layers for further improvement. Owing to the lack of ear images per person, the authors propose to transform the ear identification problem to the binary classification by composing pairwise samples and resolve it with the pairwise support vector machine (SVM). Experiments are conducted on four public databases: USTB I, USTB II, IIT Delhi I, and IIT Delhi II. The proposed method achieves promising recognition rate and exhibits decent performance compared with the state-of-the-art methods.