Color Guided Depth image denoising often suffers from the texture coping from the color image as well as the blurry effect at the depth discontinuities. Motivated by this, we propose an optimized color-guided filter for depth image denoising from different types of noises. This is a new framework that helps to mitigate the texture coping and enhance the depth discontinuities, especially in heavy noises. This framework consists of two parts namely depth driven color flattening model and patch synthesis-based Markov random field model. The first part which is a prepare step for the second part is used to mitigate the texture coping problem that faces all color guided methods. This first model consists of a modified joint bilateral filter which is used to mitigate the noise from the noisy depth image and an iterative guided bilateral filter that is proposed to flatten the colors in the color image for mitigating the texture coping problem. Based on the first part, Markov random field with an optimization technique is used for mitigating the blurry effect. Experiments indicate that our method outperforms counterpart filters with guided and non-guided manners in terms of a variety of evaluation metrics.