Object detection is a fundamental task in computer vision, consisting of both classification and localization tasks. Previous works mostly perform classification and localization with shared feature extractor like Convolution Neural Network. However, the tasks of classification and localization exhibit different sensitivities with regard to the same feature, hence the "task spatial misalignment" issue. This issue can result in a hedge issue between the performances of localizer and classifier. To address these issues, we first propose a novel Dynamic Coefficient Loss to simultaneously consider and balance the performances of classification and localization tasks. To well address anchor label misjudgment issue in irregular-shaped object detection, we define a new classification-aware IoU metric to assign anchors intelligently. Finally, we further introduce the localization factor into NMS by proposing a Classification-Localization balanced NMS. Extensive experiments on MS COCO and PASCAL VOC demonstrate that our proposals can improve RetinaNet by around 1.5% AP with various backbones.