In this paper we present a novel approach for the detection of abandoned luggage in an intelligent video surveillance system. Abandoned luggage, especially in critical infrastructures such as airports and train stations, are always dealt with as a priority security threat, and therefore an early detection is crucial for a swift reaction and evacuation initiated by the security personnel. The approach that will be presented in this paper is capable of functioning in dynamic environments and can handle luggage pieces of various sizes and colours. The novelty of this approach is in combining the data of 2 sensors, the RGB sensor and the depth sensor, to achieve a robust foreground segmentation. The novelty also lies in the use of the RGB sensor to extract the features of the suspected abandoned piece of luggage and verify them in the following frames. The tracking algorithm of the depth sensor is used to detect users and therefore eliminate persons from the list of suspected objects and reduce the amount of objects to be observed and analysed. Finally, we have evaluated the algorithm in an installation inside a fully functional tram within the frameworks of a German nationally funded project called InREAKT in three iterations.