We present the results of a study to exploit a multiple colour space model (CSM) and variable kernels for object tracking in video sequences. The basis of our work is the mean shift algorithm; for a moving target, we develop a procedure to adaptively change the CSM throughout a video sequence. The optional CSM components are ranked using a similarity distance within an inner (representing the object) and outer (representing the surrounding region) rectangle. Rather than use the standard, Epanechnikov kernel, we have also used a kernel weighted by the normalized Chamfer distance transform to improve the accuracy of target representation and localization, minimising the distance between the two distributions of foreground and background using the Bhattacharya coefficient. To define the target shape in the rectangular window, either regional segmentation or background-difference imaging, dependent on the nature of the video sequence, has been used. Experimental results show the improved tracking capability and versatility of the algorithm in comparison with results using fixed colour models and standard kernels. © Springer-Verlag Berlin Heidelberg 2007.