Drones have been widely employed in various fields, but the number of drones being used illegally and for hazardous purposes has recently increased. To prevent illegal drones, in this work, we propose a novel framework for reconstructing three-dimensional (3D) drone trajectories using a single camera. By leveraging calibrated cameras, we exploit the relationship between 2D and 3D spaces. We automatically track the drones in 2D images using a drone tracker and estimate their 2D rotations. By combining the estimated 2D drone positions with their actual length and camera parameters, we geometrically infer the 3D drone trajectories. To address the lack of public drone datasets, we also create synthetic 2D and 3D drone datasets. The experimental results show that the proposed methods accurately reconstruct drone trajectories in 3D space and demonstrate the potential of our framework for single-camera-based surveillance systems.