Separating diffuse and specular reflection components is important for preprocessing of various computer vision techniques such as photometric stereo. In this paper, we address diffuse-specular separation for photometric stereo based on light fields. Specifically, we reveal the low-rank structure of the multi-view images under varying light source directions, and then formulate the diffuse-specular separation as a low-rank approximation of the 3rd order tensor. Through a number of experiments using real images, we show that our proposed method, which integrates the complement clues based on varying light source directions and varying viewing directions, works better than existing techniques.