In this paper, we propose a framework for 3D human pose estimation with a single 360° camera mounted on the user's wrist. Perceiving a 3D human pose with such a simple setting has remarkable potential for various applications (e.g., daily-living activity monitoring, motion analysis for sports enhancement). However, no existing work has tackled this task due to the difficulty of estimating a human pose from a single camera image in which only a part of the human body is captured and the lack of training data. Therefore, we propose an effective method for translating wrist-mounted 360° camera images into 3D human poses. We also propose silhouette-based synthetic data generation dedicated to this task, which enables us to bridge the domain gap between real-world data and synthetic data. We achieved higher estimation accuracy quantitatively and qualitatively compared with other baseline methods.