Single image deraining has been widely studied in recent years. Motivated by residual learning, most deep learning based deraining approaches devote research attention to extracting rain streaks, usually yielding visual artifacts in final deraining images. To address this issue, we in this paper propose bilateral recurrent network (BRN) to simultaneously exploit rain streak layer and background image layer. Generally, we employ dual residual networks (ResNet) that are recursively unfolded to sequentially extract rain streaks and predict clean background image. Furthermore, we propose bilateral LSTMs into dual ResNets, which not only can respectively propagate deep features across multiple stages, but also bring the interplay between rain streak layer and background image layer. The experimental results demonstrate that our BRN notably outperforms state-of-the-art deep deraining networks on both synthetic datasets and real rainy images. All the source code and pre-trained models are available at https://github.com/shangwei5/BRN.