The semantic information obtained from large-scale computation in image compression is not practical. To solve this problem, we propose an Attention Aggregation Mechanism (AAM) for learning-based image compression, which is able to aggregate attention map from multiple scales and facilitate information embedding. Based on AAM and Swin Transformer, we design a Multi-scale Self-Attention Image Compression (MSAIC) approach, which is using Multi-scale Swin Transformer blocks (MSTB) and convolutional layers stacks in the down-sampling encoder and up-sampling decoder to the best embed information. Meanwhile, we use the channel-wise autoregressive entropy model for the accurate and efficient entropy probability estimation. Extensive experiments show MSAIC is effective and attain the state-of-the-art (SOTA) compression performance, also outperform the SOTA method at the high bit rate.