In this paper, a novel method to estimate the number of source signals based on support vector machine (SVM) is proposed. The estimation problem of number of source signals is transformed into the problem of multi-class classification, and the SVM is applied to solve the problem of multi-class classification. The proposed method can be used when the modulation type chooses from the QPSK and BPSK at random. The application of the novel method can expand present well-determined situation and over-determined situation into the under-determined situation. Simulation results confirmed these conclusions.