Our previous study has shown that image distortions cause saliency distraction, and that visual saliency of a distorted image differs from that of its distortion-free reference. Being able to measure such distortion-induced saliency variation (DSV) significantly benefits algorithms for automated image quality assessment. Methods of quantifying DSV, however, remain unexplored due to the lack of a benchmark. In this paper, we build a benchmark for the measurement of DSV through a subjective study. Sixteen experts in computer vision were asked to compare saliency maps of distorted images to the corresponding saliency maps of the original images. All saliency maps were rendered from ground truth human fixations. A statistical analysis is performed to reveal the behaviours and properties of human assessment of the saliency variation. The benchmark is made publicly available to the research community.