A structure-adaptive vector median filter (SAVMF) for removal of impulse noise from color images is presented in this paper. A color image is represented in quaternion form, and then quaternion Fourier transform is employed to detect the dominant orientation of the pattern in a local neighborhood. Based on the local orientation and its strength, the size, shape and orientation of the support window of vector median filter (VMF) can be adaptively computed, and thus structure-adaptive VMF is implemented. Experimental results exhibit the validity of the proposed method by showing clearly performance improvements both in noise removal and in detail preservation, compared to other VMF-based vector filters.