How to obtain higher quality reconstructed video within limited coding resources is a research focus for video coding. This paper proposes the 3D-CNN Compressed Video Quality Enhancement (3D-CVQE) network for enhancing the quality of the compressed video, as shown in Fig. 1. First, a dataset is established with 148 pairs of YUV videos that contain a large number of rich content scenes and different resolutions. According to the dataset, the “quality fluctuation”, “pixel missing”, and “position fluctuation” characteristics of compressed video are found by rethinking the video coding process. Then, based on the characteristics of “quality fluctuation” and “position fluctuation”, this paper proposes the use of non-aligned networks with multi-frame input to solve the reconstruction problem of compressed video. Specially, 3D-CNN approach is utilized which makes motion alignment not necessitated thanks to its spatial-temporal feature representation ability. Finally, based on the characteristic of “pixel missing”, this paper proposes to treat the compressed video quality enhancement task as a special video super-resolution task with no increase in resolution. It is demonstrated that the average PSNR of 18 HEVC standard sequences is enhanced 0.4652 dB at QP 37 (Low-Delay) and the number of parameters remains at 1.98 million.