In view of the observation that saliency maps generated by saliency detection algorithms usually show similarity imperfection against the ground truth, the authors propose an optimisation algorithm based on clustering and fitting (CF) for saliency detection. The algorithm uses a fitting model to represent the quantitative relationship between ground truth and algorithm-generated saliency maps. The authors use the K-means method to cluster the images into k clusters according to the similarities among images. Image similarity is measured in terms of scene and colour by using the GIST and colour histogram features, after which the fitting model for each cluster is calculated. The saliency map of a new image is optimised by using one of the fitting models which correspond to the cluster to which the image belongs. Experimental results show that their CF-based optimisation algorithm improves the performance of various single image saliency detection algorithms. Moreover, the improvement achieved by their algorithm when using both CF strategies is greater than the improvement achieved by the same algorithm when not using the clustering strategy. In addition, their proposed optimisation algorithm can also effectively optimise co-saliency detection algorithms which already consider multiple similar images simultaneously to improve saliency of single images.

Pulse coupled neural network (PCNN) is widely used in image processing because of its unique biological characteristics, which is suitable for image fusion. When combining PCNN with non-subsampled contourlet (NSCT) model, it is applied in overcoming the difficulty of coefficients selection for subband of the NSCT model. However in the original model, only the grey values of image pixels are used as input, without considering that the subjective vision of human eyes lacks the sensitivity to the local factors of the image. In this study, the improved pulse-coupled neural network model has replaced the grey-scale value of the image and introduced the weighted product of the strength of the gradient of the image and the local phase coherence as the model input. Finally, compared with other multi-scale decompositions-based image fusion and other improved NSCT-PCNN algorithms, the algorithm presented in this study outperforms them in terms of objective criteria and visual appearance.

The vision-based mobile robot's simultaneous localisation and mapping and navigation capability in dynamic environments are highly problematic elements of robot vision applications. The goal of this study is to reconstruct a static map and track the dynamic object for a camera and laser scanner system. An improved automatic calibration is designed to merge image and laser point clouds. Then, the fusion data is exploited to detect the slowly moved object and reconstruct static map. Tracking-by-detection requires the correct assignment of noisy detection results to object trajectories. In the proposed method, occluded regions are combined 3D motion models with object appearance to manage difficulties in crowded scenes. The proposed method was validated by experimental results gathered in a real environment and on publicly available data.

Single image dehazing and denoising models can simultaneously remove haze and noise with high efficiency. Here, the authors propose three variational models combining the celebrated dark channel prior (DCP) and total variations (TV) models for image dehazing and denoising. The authors firstly estimate the transmission map associated with depth using DCP, then design three variational models for colour image dehazing and denoising based on this estimation and the layered total variation (LTV) regulariser, multichannel total variation (MTV) regulariser, and colour total variation (CTV) regulariser, respectively. In order to improve the computation efficiency of the three models, the authors design their fast split Bregman algorithms via introducing some auxiliary variables and the Bregman iterative parameters. Numerous experiments are presented to compare their denoising effects, edge-preserving properties, and computation efficiencies. To demonstrate the merits of the proposed models, the authors also conduct some comparisons with several existing state-of-the-art methods. Numerical results further prove that the LTV-based model is fastest, and the CTV model is the best for denoising with edge-preserving, and it also leads to the best visually haze-free and noise-free images.

We propose a new method for human pose estimation which leverages information from multiple views to impose a strong prior on articulated pose. The novelty of the method concerns the types of coherence modelled. Consistency is maximised over the different views through different terms modelling classical geometric information (coherence of the resulting poses) as well as appearance information which is modelled as latent variables in the global energy function. Moreover, adequacy of each view is assessed and their contributions are adjusted accordingly. Experiments on the HumanEva and Utrecht multi-person motion datasets show that the proposed method significantly decreases the estimation error compared to single-view results.

The authors consider the problem of matching two shapes assuming these shapes are related by an elastic deformation. Using linearised elasticity theory and the finite-element method, they seek an elastic deformation that is caused by simple external boundary forces and accounts for the difference between the two shapes. The main contribution is in proposing a cost function and an optimisation procedure to minimise the symmetric difference between the deformed and the target shapes as an alternative to point matches that guide the matching in other techniques. The authors show how to approximate the non-linear optimisation problem by a sequence of convex problems. They demonstrate the utility of the proposed method in experiments and compare it to an iterative closest point like matching algorithm.

Non-uniformly illuminated images often suffer from low visibility in dark areas. Traditional methods usually enhance non-uniformly illuminated images by bringing out the details in the dark areas, but easily result in over-enhancement. Motivated by the Weber contrast model, the authors propose a perceptually inspired image enhancement method, which treats an image as a product of a luminance mapping (LM) transfer function and a contrast measure (CM) transfer function. The contribution of this proposed method is two-fold. Firstly, they propose a progressive LM transfer function based on the sensitivity of the human visual system to emphasise changes at low brightness level and attenuates changes at high brightness levels. Secondly, they introduce a CM transfer function, which is based on a special implementation of a neural model of the human visual receptive field, to improve local intensity contrast. Experimental comparisons with some state-of-the-art methods show that the proposed method can achieve both contrast enhancement and visual fidelity preservation.

Currently, human motion analysis using three-dimensional (3D) data creates closer awareness in computer vision with the introduction of cost-effective Kinect or other depth cameras. This study attempts to segment a continuous 3D skeletal sequence into several disjointed sub-sequences, each of which is corresponding to a complete action. To address this issue, the authors propose a supervised time-series segmentation algorithm. A bidirectional propagation search scheme is employed to reach a solution. Specifically, a human skeleton is formulated as a point in multidimensional space, and a motion trajectory is further represented as a sequence. Each training action sequence serves as an atom in a dictionary, which is adopted by an l ₂ -regularised collaborative representation classifier. Based on the fact that the reconstruction error of the collaborative representation measures the similarity between a test sub-sequence and training sequences, they utilise its variation over time to capture action transition. Cut point detection and sub-sequence recognition are simultaneously achieved. Experiments on the authors’ recorded 3D skeletal sequences demonstrate that the proposed algorithm outperforms existing human motion segmentation techniques. Their algorithm is capable of extending to segment various dimensional sequences. This extensibility is validated by synthetic signal segmentation experiments.

Due to the huge number of online videos uploaded and viewed every day, there is an emerging need nowadays for the action recognition techniques. Applying these techniques in uncontrolled and realistic videos is still a challenging task, considering the large variations in camera motion, viewpoint, cluttered background etc. Moreover, they need to be automated to be able to handle such an amount of different actions. The goal of this study is to introduce a new technique for mining mid-level discriminative patches from videos. These patches are the most representative parts that can describe an action. To achieve this goal, the authors generalise a technique borrowed from 2D images to generate bounding boxes with a high motion and appearance saliencies. Then, a clustering-classification iterative procedure is applied on the generated boxes. Then, they calculate a discriminative score for each box. Finally, they select top ranked boxes to train exemplar-SVM on low-level features which are extracted from the selected boxes. The proposed approach has been evaluated using two challenging datasets YouTube and JHMDB. The experimental results demonstrated the effectiveness of their approach to achieve a better average recognition accuracy than the state-of-the-art techniques.

Feature point detection based on convolutional neural network (CNN) has been studied widely. The effective approaches for improving detection accuracy are building a deeper network or using a multi-network cascade structure. However, some potential capacity of CNN has not been excavated. In this study, the authors mainly analyse several factors influencing CNN performance from two aspects: (i) the position relationships between feature points and (ii) the normalisation methods of coordinates. Whether the network can learn the position relationships is also studied. For extracting the deep features of images, a network containing three convolution layers is constructed. The specific geometric relationship constraints are applied during calibration to maximise the capability of the CNN for learning the position relationship between feature points. Considering that different feature points only appear in various local regions of an image, local normalisation is proposed, which increases the mapping scope of the feature points and decreases the mapping error. The experimental results prove that the specific position relationship and local normalisation obviously improve the feature point detection based on CNN. At the detection error of 5%, the average detection accuracy of eyelid feature points is improved by 7.1% and single-point detection receives a high accuracy of 97.96%.

A novel framework, named intra-class variation reduced features-based manifold regularisation dictionary pair learning model, is presented for solving facial expression recognition (FER) tasks. Since a query face and its corresponding image with intra-class variations (e.g. identity and illumination) are similar in appearance, the authors generate intra-class variation reduced features (IVRF) from the difference between a query face image and its corresponding estimated image of each expression class. IVRF can reduce negative influence from the intra-class variations and make their model robust to intra-class variations. Furthermore, a manifold regularisation term is incorporated into the dictionary pair learning model, which leads to a smoothly varying sparse representation. Their model fully takes advantage of the geometrical structure of data, which benefits the FER task. The experimental results on two public databases verify the effectiveness and superiority of their method and indicate its promising capability in expression discrimination.

When utilising non-negative matrix factorisation (NMF) to decompose a data matrix into the product of two low-rank matrices with non-negative entries, the noisy components of data may be introduced into the matrix. Many approaches have been proposed to address the problem. Different from them, the authors consider the group sparsity and the geometric structure of data by introducing -norm and local structure preserving regularisation in the formulated objective function. A graph regularised sparse NMF de-noising approach is proposed to learn discriminative representations for the original data. Since the non-differentiability of -norm increases the computational cost, they propose an effective iterative multiplicative update algorithm to solve the objective function by using the Frobenius-norm of transpose coefficient matrix. Experimental results on facial image datasets demonstrate the superiority of the proposed approach over several state-of-the-art approaches.

Palmprint gender classification can revolutionise the performance of authentication systems, reduce searching space and speed up matching rate. However, to the best of their knowledge, there is no literature addressing this issue. The authors design a new convolutional neural network (CNN) structure, fine-tuning Visual Geometry Group Network, up to 19 layers to achieve a 20-layer network, for palmprint gender classification. Experimental results show that the proposed structure could achieve good performance for gender classification. They also investigate palmprint images with 15 different kinds of spectra. They empirically find that a palmprint image acquired by the Blue spectrum could achieve 89.2% correct classification and could be considered as a suitable spectrum for gender classification. The neural network is able to classify a 224 × 224 × 3-pixel palmprint image in <23 ms, verifying that the proposed CNN is an effective real-time solution.

Transfer learning aims at adapting a model learned from source dataset to target dataset. It is a beneficial approach especially when annotating on the target dataset is expensive or infeasible. Transfer learning has demonstrated its powerful learning capabilities in various vision tasks. Despite transfer learning being a promising approach, it is still an open question how to adapt the model learned from the source dataset to the target dataset. One big challenge is to prevent the impact of category bias on classification performance. Dataset bias exists when two images from the same category, but from different datasets, are not classified as the same. To address this problem, a transfer learning algorithm has been proposed, called negative back-dropout transfer learning (NB-TL), which utilizes images that have been misclassified and further performs back-dropout strategy on them to penalize errors. Experimental results demonstrate the effectiveness of the proposed algorithm. In particular, the authors evaluate the performance of the proposed NB-TL algorithm on UCF 101 action recognition dataset, achieving 88.9% recognition rate.

The dynamic background will cause extremely negative effects on background subtraction and is difficult to eliminate. This study proposes a dynamic background subtraction method based on a spatio-temporal classification which mainly contains two key steps: temporal and spatial classifications. For temporal classification, the closest pixel sampling algorithm is used to sample background pixels in groups, which avoids centralised sampling and a complicated mathematical modelling process. For the background model obtained by group sampling, the pixels which are similar to the detected pixel are classified into the same category. According to the number of pixels in this category, the label (foreground or background) of the detected pixel can be determined thus a coarse foreground mask is obtained. For spatial classification, considering the correlation between dynamic background pixels and neighbouring pixels, a square window can be set for each foreground pixel in the coarse mask, and then all pixels in the window classified. According to the labels of these classified pixels, a more accurate foreground mask is obtained. The experiments on public datasets demonstrate that the proposed method outperforms other state-of-the-art methods.

Here, the authors present a low cost telerehabilitation system made up of a commercial red–green–blue depth (RGB-D) camera and a web-based platform. The authors goal is to monitor and assess subject movement providing acceptable and usable at-home remote rehabilitation services without the presence of a clinician. Clinical goals, defined by physiotherapists, are firstly translated into motion analysis features. A Takagi Sugeno fuzzy inference system (FIS) is then proposed to evaluate and combine these features into scores. In this stage, the ‘collaborative design’ paradigm is used in depth and complete manner: the contribution of the clinician is not limited only to the rules definition but enters in the core of the evaluation algorithm through the definition of the fuzzy rules. A case study on low back pain rehabilitation involving 40 subjects, 5 exercises, and 4 physiotherapists is then presented to the effectiveness of the proposed system. Results of the validation of the system aimed at the assessment of the reliability of the proposed approach show high correlations between clinician evaluation and FIS scores. In this scenario, due to the high correlation, each FIS could represent a virtual alter-ego of the physiotherapist which enable a real time and free second opinion.

One of the goals of person re-identification systems is to support video-surveillance operators and forensic investigators to find an individual of interest in videos acquired by a network of non-overlapping cameras. This is attained by sorting images of previously observed individuals for decreasing values of their similarity with a given probe individual. Existing appearance descriptors, together with their similarity measures, are mostly aimed at improving ranking quality. The authors address instead the issue of processing time, which is also relevant in practical applications involving interaction with human operators. They show how a trade-off between processing time and ranking quality, for any given descriptor, can be achieved through a multi-stage ranking approach inspired by multi-stage classification approaches, which they adapt to the re-identification ranking task. The authors analytically model the processing time of multi-stage system and discuss the corresponding accuracy, and derive from these results practical design guidelines. They then empirically evaluate their approach on three benchmark data sets and four state-of-the-art descriptors.

Measuring the intrinsic deformability of arbitrary small-scale subdivision of a shape is an interesting meanwhile valuable research topic. Such measurement can be directly utilised as a reliable criteria to partition shape into small components and then assist in shape modelling and description. Compared to global modelling, through constructing subdivision-based complex shape description, the accuracy and flexibility of shape representation can be significantly improved. In this study, the authors propose a line segment advection (LSA)-based vertex-level three-dimensional shape deformability measuring method. It can highlight the deformability characteristics of each shape part in any scale and size. The measurement is realised mainly based on the advection of line segments connecting neighbouring shape mesh vertices. For 3D shapes, since the line segment of triangular mesh facet directly reflects the minimal neighbourhood relationships and mesh microstructure, its advection can capture the finest details of shape deformability. Then, after transferring that information into neighbouring vertices, a vertex-level shape deformability measurement can be acquired. Besides, to demonstrate the value of the proposed measuring method to shape partitioning and piecewise shape modelling, a straightforward shape partitioning method is introduced as well. Extensive experiments on three publicly available databases are conducted to verify the effectiveness of proposed methods.

Automatic detection and tracking of a small target in infrared (IR) images are of great importance. Toggle operator (TO) is the newest class of non-linear operator morphology that has been widely used in detection and tracking the target in IR images. The most important problem in improving the efficiency of the TO is to use structural elements (SEs) in accordance with signal-to-clutter ratio (SCR) of each image. Generally, the clutters and targets are different in case of each image; therefore, for images with different SCRs, using SEs with fixed pixels and dimensions cannot lead to successful target detection. In this study, a new method is presented based on genetic algorithm to achieve adaptive SE for target detection in IR images. In this method, by designing the SE in accordance with the characteristics of each image, a large amount of background clutter and noise is suppressed and the contrast between target and background is increased. The results of a large set of real IR images including moving targets show that the proposed algorithm is effective in target detection. In the proposed method, the contrast between the target and background clutter is greatly increased while maintaining a low false alarm rate.

In this study, the authors propose a variational approach based on total generalised variation (TGV) and local gradient information to fuse multi-focus images as well as medical images of computed tomography and magnetic resonance. They use the second-order TGV as the regularisation term and local gradient information as the fusion weight to extract image features. To compute the new model effectively, the primal-dual algorithm is carried out. Various experiments are made to verify the effectiveness of the proposed methods.

The multiplicative noise removal problem has received considerable attention recently. To solve this problem, various variational models have been proposed, which minimise an energy functional composed of the data term and the regularisation term. Regarding the regularisation term, a first-order model is frequently used to remove multiplicative noise, which may cause staircase effect and loss of contrast in the output image. In this study, the authors use a second-order model, the total curvature (TC), to solve the above problem. The TC model has the benefit of removing the staircase effect and maintaining image edges, contrasts and corners. The augmented Lagrange method is utilised to solve the proposed TC model by introducing auxiliary variables, Lagrange multipliers and using alternating optimisation strategy. In each loop of optimisation, the fast Fourier transform, generalised soft threshold formulas, projection method and gradient descent method are integrated effectively. The experimental results show that the TC model can effectively remove staircase effect and preserve smoothness, via comparison with the first-order model (total variation regularisation and Perona–Malik regularisation). Furthermore, the TC model is better than another second-order model based on bounded Hessian regularisation in preserving contrast and corner.

3D human body parameters measurement is a challenging task due to two main reasons: (i) it is difficult to reconstruct 3D human model due to flexible deformation of non-rigid body during images capturing process and (ii) there lies a gap between 3D model and body parameters. To address these two issues, a 3D human body parameters measurement system is represented. With the object freely spinning in front of a Kinect, body parameters are calculated. To reduce registration errors caused by body deformation while rotating, a piecewise tracking and mapping algorithm based on KinectFusion framework is proposed. Then model–model iterative closest point and non-rigid constraints are introduced to optimise alignments and disambiguate different surfaces caused by aliasing in the piecewise strategy. Finally, a novel method is presented to measure the volume and perimeter of human body with the truncated signed distance function values of voxels. Extensive experimental results show that the proposed method achieves comparable accuracy to the state of the arts, and the error of volume and perimeter measurements are 2.0 and 5.8%, respectively.