The bilateral filter is popularly used for image enhancement. By using a range kernel along with a spatial kernel, the filter is able to smooth images without excessive blurring of edges. It has been shown that the enhancement capacity of the filter can be boosted by adapting the width of the range kernel at each pixel. A fast algorithm for grayscale images was recently proposed for this so-called adaptive bilateral filter, which is otherwise computationally expensive. This can trivially be extended for color filtering using channelwise processing. However, developing an efficient algorithm that can exploit correlations between color channels is not straightforward. We show that such a fast algorithm can be developed by first expressing the filtering in terms of the local histogram and then approximating the latter by an uniform distribution. The distribution in question is along the direction of maximum variance in the RGB space, which we compute from the local covariance (this is done efficiently using power iterations). The local covariances in turn are computed using fast convolutions. To demonstrate the effectiveness of our fast algorithm, we apply it for sharpening, detail enhancement, and deblocking.