Here, the authors have extensively investigated the unconstrained ear recognition problem. The authors have first shown the importance of domain adaptation, when deep convolutional neural network (CNN) models are used for ear recognition. To enable domain adaptation, the authors have collected a new ear data set using the Multi-PIE face data set, which they named as Multi-PIE ear data set. The authors have analysed in depth the effect of ear image quality, for example, illumination and aspect ratio, on the classification performance. Finally, the authors have addressed the problem of data set bias in the ear recognition field. Experiments on the UERC data set have shown that domain adaptation leads to a significant performance improvement. For example, when VGG-16 model is used and the domain adaptation is applied, an absolute increase of around 10% has been achieved. Combining different deep CNN models has further improved the accuracy by 4%. In the experiments that the authors have conducted to examine the data set bias, given an ear image, they were able to classify the data set that it has come from with 99.71% accuracy, which indicates a strong bias among the ear recognition data sets.