Gesture is a basic human feature and an indispensable part of interpersonal communication. The development of gesture recognition technology has made it possible for humans to interact with machines and other devices. The millimeter-wave radar can detect range, speed, and angle information during gesture movement, thereby achieving the purpose of identifying different gestures with stronger environmental adaptability. In this paper, we use the AWR1443 millimeter wave radar developed by Texas Instruments to collect gesture signal. According to the characteristics of the signal, the signal processing algorithms such as FFT are used to extract the range and Doppler frequency shift information. Then the range and Doppler frequency shift information are respectively coupled with time to form the Range-Time Matrix (RTM), Doppler-Time Matrix (DTM), and Range-Doppler Matrix (RDM). Based on this, a data set for gesture recognition is established, and the purpose of gesture recognition is achieved by training neural network. We design a RD-T network to make full use of RTM and DTM information. In the end, we replace the MLP in RD-T network with the LSTM to compare performance of LSTM and RD-T.