Recently many efforts have been devoted to learning non-linear predictions from neighboring samples with deep neural networks. However, existing methods mainly generate predictions with local reference samples, regardless of nonlocal self-similarity.
In this paper, we aim to incorporate local and nonlocal correlations for intra prediction and propose a two-stage coarse-to-fine network (CTFN), which is integrated into VVC codec as an optional intra prediction mode. The prediction process of CTFN is decomposed into two stages. In the first stage, we train a set of networks to generate a coarse result with local reference samples. In the second stage, we extract sufficient features from nonlocal region using the coarse result as priors and transform the features into a fine prediction result. In particular, a patch-wise attention layer (PAL) is designed in the second stage that can fully explore nonlocal correlations in feature domain and assign weights to each nonlocal feature adaptively, as shown in Fig. 1. As such, the proposed CTFN can not only learn a non-linear mapping from local context, but also explicitly borrow similar features from nonlocal region in a weighted form.