The paper presents the development of an intelligent image processing algorithm capable of detecting fatigue defects from images of the rail surface. The links between the defect detection algorithm and 3D models for rail crack propagation are investigated, considering the influence of input parameters (materials, vehicle characteristics, loading conditions). The dynamic behaviour at the wheel-rail interface resulting in contact forces responsible for stressing and straining the rail material are imported from vehicle dynamics simulations. The integration of the simulated results from vehicle dynamics, contact and fracture mechanics models offer more reliable estimation of the stress intensity factors (SIF). Also the sensitivity analysis related to materials, vehicle characteristics, and loading conditions will provide further understanding of the factors that influence crack propagation in rails such as shear stresses, hydraulic pressure, fluid entrapment and squeeze film effect. This novel application of image processing for the detection of rail surface rolling contact fatigue (RCF) damage and automatic incorporation in a crack growth model represents an important contribution to the development of modern techniques for non-destructive rail inspection. This will result in improved planning/scheduling of future rail maintenance (e.g. rail grinding, renewal), less disruptions and reduced track maintenance costs in rail industry.