There is larger compression potential for surveillance video coding due to the inherent superredundancy in fixed camera scenarios, which is not fully utilized in the general-purpose MPEG-like video coders. In this paper, we propose a novel compression framework for indoor surveillance video in which the background and foreground humans are separately compressed. Since the indoor surveillance video’s background is static, the background is extracted and shared between frames. The foreground humans are extracted and represented with compact features, which are used to reconstruct the foreground humans using a conditional generation network. Additionally, we also use two optimization methods, compensation of residuals and quality enhancement network, for improving the reconstructed image quality. With aid of compact features of semantic humans and static background, this work delivers a greatly improved compression ratio and appealing visual quality. Experimental results show that our method can save approximately 52.08% bit-rates compared with HEVC.