Matrix factorization based hashing has been very effective in addressing the cross-modal retrieval task. In this work, we propose a novel supervised hashing approach utilizing the concepts of matrix factorization which can seamlessly incorporate the label information. In the proposed approach, the latent factors for each individual modality are generated, which are then converted to the more discriminative label space using modality specific linear transformations. In the first stage of the approach, the hash codes are learned using an alternating minimization algorithm and in the next stage, modality specific hash functions are learned to convert the original features of the cross-modal data into the hash code domain. In addition, we also propose an extension of the approach for handling very large amounts of data during the training stage. Extensive experiments performed on the single label Wiki, and the multi-labeled MirFlickr and NUS-WIDE datasets show the effectiveness of the proposed approach.