Hyperspectral videos contain images with a large number of light wavelength indexed bands that can facilitate material
identification for object tracking. Most hyperspectral trackers use hand-crafted features rather than deep learning gener-
ated features for image representation due to limited training samples. To fill this gap, this paper introduces a band atten-
tion aware ensemble network (BAE-Net) for deep hyperspectral object tracking, which takes advantages of deep models
trained on color videos for feature representation. Specifically, an autoencoder-like band attention block is introduced
to learn the dependencies among bands and generate bandwise weights. Guided by these weights, hyperspectral im-
ages are then divided into a number of three-channel images. These three-channel images are fed into a deep color track-
ing network, producing several weak trackers. Finally, weak trackers are fused using ensemble learning for target location.
Experimental results on hyperspectral datasets show the effectiveness and advantages of the proposed deep hyperspectral tracker.