This paper presents a novel Time-of-Flight (ToF) depth recovery algorithm minimizing a new quadratic energy function utilizing depth and infrared data. The proposed energy function consists of a filtering term and a reconstruction term to remove noise and fill holes simultaneously in a depth image. In the filtering term, a new multilateral weight is introduced by fully using available spatial, depth, and infra¬red information. In the reconstruction term, a Poisson equation for reconstructing a depth image from its gradients is used whereas the depth gradients inside hole regions are interpolated with the proposed infrared-guided moving least squares. The recovered depth data can be obtained by solving a sparse linear system derived from minimizing the proposed energy function. Experimental results demonstrate that the proposed algorithm provides good depth recovery performance without introducing texture copy or blur artifacts com¬pared to the existing methods.