Human activity classification is a critical part of intelligent human-computer interaction, which is promising in various applications. The radar system provides a complementary input when the visible light cannot work. It can effectively tackle darkness, occlusion, and non-line-of sight conditions. The frequency modulation induced by the micromotion of human movements can be captured and utilized by radar systems. Currently, the integration of deep learning methods and classical signal processing has been more and more prominent in the radar-based human behaviour sensing field. In this work, we propose a compact deep learning model to classify various human activities by the corresponding frequency modulation. Considering most deep neural networks are both data-hungry and resource-hungry, we integrate transfer learning and network pruning techniques to reduce the number of labelled training samples and computational burden. The experiments demonstrate that our method not only outperforms convolutional neural networks trained from scratch but also significantly slims its model size and computing operations.