Existing methods for cross-component prediction focus on the chroma intra prediction but neglect the chroma prediction across multiple frames. In this paper, we propose a novel compression framework that leverages chroma frame sampling and inter-frame chroma prediction to address this problem. Specifically, chroma frame sampling discards the chroma components in inter-predicted frames to further reduce the bit consumption, whereas inter-frame chroma prediction recovers the discarded chroma components with optical flow estimation and post-training optimization for a guaranteed fidelity. To our best knowledge, this paper is the first attempt to achieve deep learning-based inter prediction of chroma components. To accommodate the proposed framework, we redesign the HEVC codec to enable hybrid coding of luma and chroma components. Experimental results show that the proposed framework achieves up to 0.76% BD-rate reduction when compared to standard HEVC.