We propose a novel video saliency detection method based on pairwise interaction learning in this paper. Different from the traditional video saliency detection methods, which mostly combine spatial and temporal features, we adopt Least squares Conditional random field (LS-CRF) to capture the interaction information of regions within a frame or between video frames. Specifically, dual graph-connection models are built on superpixels structure of each frame for training and testing, respectively. In order to extract the essential scene structure from video sequences, LS-CRF is introduced to learn the background texture, object components and the various relationships between foreground and background regions through the training set, and each region will be distributed an inferred saliency value in testing phase. Benefitting from the learned diverse relations among scene regions, the proposed approach achieves reliable results especially on multiple objects scenes or under highly complicated scenes. Further, we substitute weak saliency maps for pixel-wise annotations in training phase to verify the expansibility and practicability of the proposed method. Extensive quantitative and qualitative experiments on various video sequences demonstrate that the proposed algorithm outperforms conventional saliency detection algorithms.