Deep hierarchical models for feature learning have emerged as an effective technique for object representation and classification in recent years. Though the features learnt using deep models have shown lot of promise towards achieving invariance to data transformations, this primarily comes at the expense of using much larger training data and model size. In the proposed work we devise a novel technique to incorporate rotation invariance, while training the deep model parameters. The convolution weight parameters in the network architecture are constrained to exhibit circular symmetry resulting in ``rotation equivariance'' of output feature maps. Rotation invariance is further achieved by max-pooling of the feature maps later in the hierarchy. We also show that by incorporating circular symmetry constraint into the training loss function, rotation invariance can be achieved with-in deep neural network framework with much lesser training data and model parameters. Our experiment results evaluated on rotated MNIST dataset further objectively validate the contribution.