Image smoothing is a very important topic in image processing. Among these image smoothing methods, the $L_0$ gradient minimization method is one of the most popular ones. However, the $L_0$ gradient minimization method suffers from the staircasing effect and over-sharpening issue, which highly degrade the quality of the smoothed image. To overcome these issues, we use not only the $L_0$ gradient term for finding edges, but also a surface area based term for the purpose of smoothing the inside of each region. An alternating minimization algorithm is suggested to efficiently solve the proposed model, where each subproblem has a closed-form solution. Leveraging the introduced surface area term, the proposed method can effectively alleviate the staircasing effect and the over-sharpening issue. The superiority of our method over the state-of-the-art methods is demonstrated by a series of experiments.