Cascade Deformable Part Models (DPMs) are cascade frameworks to speed up Deformable Part Models (DPMs), which are one of the state-of-the-art solutions for object detection. Its idea is to reject most non-object hypotheses from the early stages of detection process. By investigating the dependency between hypotheses over scales, we introduce a novel pruning method to accelerate Cascade DPM frameworks. Our scale pruning method includes two following strategies: a) rejecting the neighboring scales if the score at a certain scale is too low; b) keeping a few scales with the highest scores, instead of storing all scales per location. Next, we extend this pruning technique to the 3D neighborhood pruning and describe a novel approach to evaluate the root score efficiently without full computation as existing cascade DPM methods. Finally, look-up tables are introduced to work with flexible neighborhood whose volume varies over hypotheses. As a result, our cascade model is equipped with an efficient and aggressive pruning mechanism. Extensive experiments reveal that the proposed method is faster than the state-of-the-art methods for both problems of object detection and face detection.