Current video classification approaches suffer from catastrophic forgetting when they are retrained on new databases.
Continual learning aims to enable a classification system with learning from a succession of tasks without forgetting.
In this paper we propose to use a transformer-based video class incremental learning model. During a succession of
learning steps, at each training time, the transformer is used to extract characteristic spatio-temporal features from videos
corresponding to a set of classes. When new video classification tasks become available, we train new classifier modules
with the transformer-extracted features, gradually building a mixture model. The proposed methodology enables continual
class learning in videos without being required to consider the learning of an initial set of classes, leading to low computation and memory requirements.
The proposed model is evaluated on standard action recognition datasets including UCF101 and HMDB51, which are split into sets of classes, to be learnt sequentially.
Our proposed method significantly outperforms the baselines on all datasets.