Viewpoint estimation in a 2D image is a challenging task due to the great variations in the object’s shape, appearance,
visible parts, etc. To overcome the above difficulties, a new deep neural network is proposed, which employs the key-points of the object as a regularization term and a semantic bridge connecting the raw pixels with the object’s viewpoint. A series of Hourglass structures are adopted for key-point
extraction. With the extracted key-points, an LSTM based network is designed to model both the intrinsic relationship among the key-points and the underlying connections between the key-points and the object’s viewpoint. A multitasks learning scheme is designed to optimize the key-point detection and the viewpoint estimation performance simultaneously. The experiment results on the PASCAL 3D+ dataset have demonstrated the effectiveness of the proposed approach.