This paper presents an efficient convolutional neural net- work (CNN)-based multiple path search (MPS) algorithm to detect multiple spatial-temporal action tubes in videos. With the pass information and the accumulated scores generated by forward message passing, the new algorithm reuses these information to simultaneously find multiple paths in back- ward path tracing without repeating the search process. More- over, to rectify the potentially inaccurate bounding boxes, we also propose a video localization refinement scheme to further boost the detection accuracy. Simulations show that the pro- posed algorithm provides competing performance compared with the main state-of-the-art works on the widespread UCF- 101 dataset with yet lower complexity.