In-Loop filter is a key part in High Efficiency Video Coding(HEVC) which effectively removes the compression artifacts.Recently, many newly proposed methods combine residual learning and dense connection to construct a deeper network for better in-loop filtering performance. However,the long-term dependency between blocks is neglected, and information usually passes between blocks only after dimension compression. To address these issues, we propose the Progressive Rethinking Block (PRB) to deliver long-term memory between the neighboring blocks and allow information to flow without compression, which is similar to human decision mechanism – usually reviewing the complete past memorized experiences to decide in the present, not just based on simple principles summarized before. PRBs further establish the Progressive Rethinking Network (PRN). In addition, we calculate the Multi-scale Mean value of Coding Units (MM-CU) to generate the side information maps which guide the training of the network by novelly telling the network architecture of the entire coding partition tree. Experimental results show that our proposed partition tree guided PRN provides 9.6% BD-rate reduction on average compared to the HEVC baseline.