Visual tracking in unconstrained environments often involves following an object that exhibits a number of appearance changes from factors such as scale change, rotation, and illumination. Effective tracking requires adapting a tracker to the object’s changing appearance over time. When a target becomes occluded by other objects in the scene, a naive tracker may end up learning the appearance of the occluding object. Our work introduces a method of detecting occlusion by considering the color profile of the target to prevent inappropriate tracker updates while the target is occluded. We show improved overlap and central location precision with three visual trackers when adding our hue-based occlusion detection to each tracking system.