We propose sandwiched video compression – a video compression
system that wraps neural networks around a standard video codec.
The sandwich framework consists of a neural pre- and post-processor
with a standard video codec between them. The networks are trained
jointly to optimize a rate-distortion loss function with the goal of significantly improving over the standard codec in various compression
scenarios. End-to-end training in this setting requires a differentiable
proxy for the standard video codec, which incorporates temporal
processing with motion compensation, inter/intra mode decisions,
and in-loop filtering. We propose differentiable approximations to
key video codec components and demonstrate that, in addition to
providing meaningful compression improvements over the standard
codec, the neural codes of the sandwich lead to significantly better rate-distortion performance in two important scenarios. When
transporting high-resolution video via low-resolution HEVC, the
sandwich system obtains 6.5 dB improvements over standard HEVC.
More importantly, using the well-known perceptual similarity metric,
LPIPS, we observe 30% improvements in rate at the same quality
over HEVC. Last but not least, we show that pre- and post-processors
formed by very modestly-parameterized, light-weight networks can
closely approximate these results