Recently, vulnerable samples have been shown to be crucial
for improving adversarial training performance. Our analysis
on existing vulnerable samples mining methods indicate that
existing methods have two problems: 1) valuable connections
among different pairs of natural samples and their adversarial
counterparts are ignored; 2) parts of vulnerable samples are
unconsidered. To better leverage vulnerable samples, we propose INter PAir ConstrainT (INPACT) and Vulnerable Aware
adveRsarial Training (VART) to address these drawbacks respectively. INPACT assesses adversarial risk with more comprehensive regularization on sample relationships, which takes
both inter and intra connections of natural/adversarial sample
pairs into consideration. Meanwhile VART makes full use of
all vulnerable samples, including notable proportion neglected
by existing instance re-weighting strategies. Extensive experiments on different datasets and backbones demonstrate the
effectiveness of the proposed method.