The Integrated square error (ISE), as a robust criterion for measuring the difference of densities between two datasets, have been commonly used in pattern recognition. In this paper, two different criteria for evaluating candidate feature subsets are investigated: first, a novel supervised feature selection criterion based on ISE and random permutation of a single feature is proposed, which presents a feature ranking criterion to measure the importance of each feature by computing the ISE over the feature space. Second, a synergetic feature selection criterion is developed. Experimental results on synthetic and real data set show the superior or at least comparable performance compared with existing feature selection algorithms.