In this work, we propose a person segmentation system that achieves high segmentation accuracy with a much smaller CNN network. In this approach, key-point detection annotation is incorporated for the first time and a novel spatial saliency map, in which the intensity of each pixel indicates the likelihood of forming a part of the human and reflects the distance from the body, is generated to provide more spatial information. Additionally, a LightWeight automatic Person Segmentation Network (LWPSN) is proposed, which is small and efficient for person segmentation by leveraging atrous convolution.