The recent trend towards miniaturization of mobile projectors is allowing new forms of information presentation and interaction. Projectors can easily be moved freely in space either by humans or by mobile robots. This paper presents a technique to dynamically track the orientation and position of the projection plane only by analyzing the distortion of the projection by itself, independent of the presented content. It allows distortion-free projection with a fixed metric size for moving projector-camera systems. To do this, an optical flow-based model is extended to the geometry of a projector-camera unit. Solving an overdetermined system of equations on pixel level leads to the pose offset between two images. In order to reach a high invariance to illumination changes we use adaptive edge images. Image pyramids allow a fast pose estimation. Because of the global optimization, there is no dependence of the availability of local feature points.