Performance of 6DoF pose estimation techniques from RGB/RGB-D images has improved significantly with sophisticated deep learning frameworks. These frameworks require large-scale training data based on real/synthetic RGB/RGB-D information. Difficulty of obtaining adequate training data has limited the scope of these frameworks for ubiquitous application areas. Also, fast pose estimation at inference time often requires high-end GPU(s) that restricts the scope for its application in mobile hardware. To address the requirement of training data adequacy, we propose a novel domain adaptation strategy to train from textureless CAD models with synthetic depth information only, and facilitate inferring poses from RGB images only. To allow faster inference on mobile hardware, we propose two lightweight architectures with a trade-off between ease of training and performance required by different applications. Experiments show comparable performance to the state-of-the-art in the challenging T-LESS dataset, with an inference time of $\sim200$ ms using CPU on Google Pixel 2.