The capabilities of biometric systems have recently made extraordinary leaps by the emergence of deep learning. However, due to the lack of enough training data, the applications of the deep neural network in the ear recognition filed have run into the bottleneck. Moreover, the effect of fine-tuning from some pre-trained models is far less than expected due to the diversity among different tasks. Therefore, the authors propose a large-scale ear database and explore the robust convolutional neural network (CNN) architecture for the ear feature representation. The images in this USTB-Helloear database were taken under uncontrolled conditions with illumination, pose variation and different level of ear occlusions. Then they fine-tuned and modified some deep models on the proposed database through the ear verification experiments. First, they replaced the last pooling layers by spatial pyramid pooling layers to fit arbitrary data size and obtain multi-level features. In the training phase, the CNNs were trained both under the supervision of the softmax loss and centre loss to obtain more compact and discriminative features to identify unseen ears. Finally, three CNNs with different scales of ear images were assembled as the multi-scale ear representations for ear verification. The experimental results demonstrate the effectiveness of the proposed modified CNN deep model.