Colonoscopy video frames might be contaminated by bright spots with unsaturated values known as specular reflection. Detection and removal of such reflections could enhance the quality of colonoscopy images and facilitate diagnosis procedure. In this paper, we propose a novel two-phase method for this purpose, consisting of detection and removal phases. In the detection phase, we employ both HSV and RGB color space information for segmentation of specular reflections. We first train a non-linear SVM for selecting a color space based on statistical image features extracted from each channel of the color spaces. Then, a cost function for detection of specular reflections is introduced. In the removal phase, we propose a two-step inpainting method which consists of appropriate replacement patch selection and removal of the blockiness effects. The proposed method is evaluated by testing on an available colonoscopy image database where accuracy and Dice score of 99.68% and 71.79% are achieved respectively.