Temporal Action localization is a more challenging vision task than action recognition because videos to be analyzed
are usually untrimmed and contain multiple action instances. In this paper, we investigate the potential of recurrent neural network, toward three critical aspects for solving this problem, namely, high-performance feature, high-quality temporal segments and effective recurrent neural network architecture. First of all, we introduce the two-stream (spatial and temporal) network for feature extraction. Then, we propose a novel temporal selective search method to generate temporal segments with variable lengths. Finally, we design a twobranch LSTM architecture for category prediction and confi- dence score computation. Our proposed approach to action localization, along with the key components, say, segments generation and classification architecture, are evaluated on the THUMOS’14 dataset and achieve promising performance by comparing with other state-of-the-art methods.