Uncertain motion of typical surveillance targets, e.g. slow moving or stopped, abrupt acceleration, and uniform motion makes a single salient motion detection algorithm unsuitable for accurate segmentation. It becomes even more challenging in case of the camera is non-stationary. In this paper, first, a simple local adaptive temporal differencing method is proposed to detect moving objects boundaries and partial interiors. To improve the accuracy of detection, a Bottom-up Variable Block Size block matching method is employed to identify the existence of possible moving object blocks and then an adaptive Kalman filter is applied to distinguish salient motions from other distracting motions. At last, the motion data from two algorithms are successfully fused to determine whether a region has been changed or not. Experimental results comparing the proposed and other competing methods are evaluated objectively and show that the proposed method achieves promising motion results for a variety of real environments.