Scene understanding is an active area of research in computer vision that encompasses several different problems. The LiDARs and stereo depth sensor have their own restrictions such as light sensitiveness, power consumption and short-range. In this paper, we propose a two-stream deep adversarial network for single image depth estimation in RGB images. For stream I network, we propose a novel encoder-decoder architecture using residual concepts to extract course-level depth features. Stream II network purely processes the information through the residual architecture for fine-level depth estimation. Also, we designed a feature map sharing architecture to share the learned feature maps of the decoder module of stream I. Sharing feature maps strengthen the residual learning to estimate the scene depth and increase the robustness of the proposed network. A benchmark NYU RGB-D v2 database is used to evaluate the proposed network for single image depth estimation. Both qualitative and quantitative analysis has been carried out to analyze the effectiveness of the proposed network for scene depth prediction. Performance analysis shows that the proposed method outperforms other existing methods for single image depth estimation.