The prediction drifts to the non-object backgrounds is a critical issue in conversional correlation filter (CF) based trackers. The key insight of this paper is to propose a doublelayer model to address this problem. Specifically, the first layer is a CF tracker, which is employed to predict a rough position of the target, and the objectness layer, which is regarded as the second layer, is utilized to reveal the object characteristics of the predicted target. The novel objectness layer firstly constructs a set of target-related object proposals, which satisfy both the spatial and temporal constraints. And then an objectness classifier is learned upon the proposal set to best separate the target from the noise background proposals. The convincing experimental results on the challenging OTB100 and TC128 dataset demonstrate the effectiveness of the presented approach.