Face super-resolution has been studied for decades, and many approaches have been proposed to upsample low-resolution face images using information mined from paired low-resolution (LR) images and high-resolution (HR) images. However, most of this kind of works only simply sharpen the blurry edges in the upsampled face images and typically no photo-realistic face is reconstructed in the final result. In this paper, we present a GAN-based algorithm for face super-resolution which properly synthesizes photo-realistic super-recovered face. To this end, we introduce semi-dual optimal transport to optimize our model such that the distribution of its generated data can match the distribution of a target domain as much as possible. This way would endow our model with learning the mapping of distribution from unpaired LR images and HR images with desired properties. We demonstrate the robustness of our algorithm by testing it on Color FERET database and show that its performance is considerably superior to all state-of-the-art approaches.