Head pose estimation from depth image is a challenging problem, considering its large pose variations, severer occlusions, and low quality of depth data. In contrast to existing approaches that take 2D depth image as input, we propose a novel deep regression architecture called Head PointNet, which consumes 3D point sets derived from a depth image describing the visible surface of a head. To cope with the non-stationary property of pose variation process, the network is facilitated with an ordinal regression module that incorporates metric penalties into ground truth label representation. The soft label representation encodes inter-class and intra-class information contained in the class labels simultaneously, and guides the network to learn discriminative features. Experiments on two challenging datasets, namely the Biwi Head Pose Dataset and Pandora Dataset, show that our proposed method outperforms state-of-the-art approaches.