Feature Pyramid Network (FPN) plays a critical role and is indispensable for object detection methods. In recent years, attention mechanism has been utilized to improve FPN due to its excellent performance. Existing attention-based FPN methods generally work with a complex structure, resulting in an increase of computational costs. In view of this, we propose a novel Channel Self-Attention Guided Feature Pyramid Network (CAG-FPN), which not only has a simple structure but also consistently improves detection accuracy. We observe that introducing channel self-attention to the features at the highest level is helpful for object detection, since modeling long-range dependencies between channels triggers an implicit clustering of the same categories of objects, enhancing the semantic continuity. Moreover, our CAG-FPN can be readily plugged into both one-stage and two-stage FPNbased detectors. Experiments on MS COCO dataset verify the superiority and generalization ability of our CAG-FPN. Code is available at https://github.com/ZY-IMU-CV/CAGFPN_CJ_2023.