Image aesthetic assessment is important for finding well taken and appealing photographs but is challenging due to the ambiguity and subjectivity of aesthetic criteria. We develop the pairwise aesthetic comparison network (PAC-Net), which consists of two parts: aesthetic feature extraction and pairwise feature comparison. To alleviate the ambiguity and subjectivity, we train PAC-Net to learn the relative aesthetic ranks of two images by employing a novel loss function, called aesthetic-adaptive cross entropy loss. Then, we develop simple schemes for using PAC-Net in the tasks of aesthetic ranking and aesthetic classification, respectively. Experimental results demonstrate that PAC-Net achieves the state-of-the-art performances in both the ranking and classification applications.