We introduce ImportantAug, a technique to augment training data
for speech classification and recognition models by adding noise
to unimportant regions of the speech and not to important regions.
Importance is predicted for each utterance by a data augmentation
agent that is trained to maximize the amount of noise it adds while
minimizing its impact on recognition performance. The effectiveness
of our method is illustrated on version two of the Google Speech
Commands (GSC) dataset. On the standard GSC test set, it achieves
a 23.3% relative error rate reduction compared to conventional noise
augmentation which applies noise to speech without regard to where
it might be most effective. It also provides a 25.4% error rate reduction
compared to a baseline without data augmentation. Additionally,
the proposed ImportantAug outperforms the conventional noise augmentation
and the baseline on two test sets with additional noise
added.