This paper introduces the application of a visual relationship classifier as a standalone system that is meant to be used with external detectors. Through these lens, we propose a training scheme that uses unannotated pairs of objects as negative samples in order to improve precision. The proposed network architecture incorporates common techniques presented in related state-of-the-art solutions with a novel positional encoding scheme. We evaluate the proposed training method and architecture on the Open Images dataset and improve mAP from 34.6% to 78.2% when considering all possible object pairings in each image. For a case where only ground-truth pairs are considered, our method presents a small decrease, from 91.0% to 88.8% mAP.