The task of head pose estimation from a single depth image is challenging, due to the presence of large pose variations, occlusions and inhomegeneous facial feature space. To solve the problem, we propose Deep Regression Forest with Soft-Attention (SA-DRF) in a multi-task learning setup. It can be integrated with a general feature learning net and jointly learned in an end-to-end manner. The soft-attention module is facilitated to learn soft masks from the general features and feeds the forest with task-specific features to regress head poses. Experiments on the Biwi Head Pose and Pandora datasets demonstrate its superior performance compared to current state-of-the-arts.