A sub-pixel disparity refinement algorithm based on Lagrange interpolation is proposed to calculate the sub-pixel disparity and the elevation of the target construction on the remote sensing image. The accurate integer matching feature points are obtained by improved affine scale invariant feature transform. A series of interpolation samples can be estimated through the three times magnification results of each feature matching point and its close neighbor pixels. The final sub-pixel deviation can be calculated by the interpolation samples. Experiment results show that a more accurate elevation information can be obtained on the remote sensing image with corrective epipolar line, and an accurate sub-pixel disparity result can be estimated without the epipolar line constraint as well.