Audio processors whose parameters are modified periodically
over time are often referred as time-varying or modulation based
audio effects. Most existing methods for modeling these type of
effect units are often optimized to a very specific circuit and cannot
be efficiently generalized to other time-varying effects. Based on
convolutional and recurrent neural networks, we propose a deep
learning architecture for generic black-box modeling of audio processors
with long-term memory. We explore the capabilities of
deep neural networks to learn such long temporal dependencies
and we show the network modeling various linear and nonlinear,
time-varying and time-invariant audio effects. In order to measure
the performance of the model, we propose an objective metric
based on the psychoacoustics of modulation frequency perception.
We also analyze what the model is actually learning and how the
given task is accomplished.