Optical coherence tomography (OCT) is a powerful method for imaging the retinal layers. In this paper, we develop a novel 3D fully convolutional deep architecture for automated segmentation of retinal layers in OCT scans. This model extracts features from both the spatial and the inter-frame dimensions by performing 3D convolutions, thereby capturing the information encoded in multiple adjacent frames. The proposed network is based on an encoder-decoder framework in which the convolution layers are interlaced with pooling layers in the encoder and with unpooling layers in the decoder, respectively. Consequently, a hierarchy of shrinking 3D feature maps are learned in the encoder and enlarged to the size of original input image for semantic segmentation in the decoder. The framework is validated on thirteen 3D OCTs captured by the Topcon 3D OCT with comparisons against two state-of-the-art segmentation methods including one recent 2D deep learning based approach to substantiate its effectiveness.