Action recognition in top-view 360° videos is an emerging research topic in computer vision. Existing work utilizes a global projection method to transform 360° video frames to panorama frames for further processing. However, this unwrapping suffers from a problem of geometric distortion i.e., people present near the centre in the 360° video frames appear highly stretched and distorted in the corresponding panorama frames (observed in 37.5% of the total panorama frames in 360Action dataset). Thus, recognizing the actions of people who are near the centre becomes difficult, thereby affecting the overall action recognition performance. In this work, we overcome the above challenge by utilizing distortion-free person-centric images of the persons near the centre, extracted directly from the input 360° video frames. We propose a simple yet effective hybrid two-stream architecture consisting of a panorama stream and a person-centric stream where outputs from both streams are combined together to detect the overall actions in a video. We perform experiments to validate the efficacy of the proposed method on the recently introduced 360Action dataset and achieve an overall improvement of 2.3% mAP compared to the state-of-the art method and a maximum improvement of 22.7% AP for pickup action, which happens mostly near the centre.