A solution to deploy object detection systems to practical applications is to build cascade frameworks which do threshold comparisons in each stage to efficiently discard a large number of negative objects. For particular applications, these thresholds should be retrained for better effectiveness and the efficiency via training datasets. It means that we have to store labeled datasets permanently or collect huge data (for highquality thresholds) whenever learning new thresholds. Both approaches are inconvenient and expensive in terms of memory and data collection cost. In this paper, we propose a novel threshold selection mechanism, named Adaptive Cascade Threshold Learning (ACTL), which learns thresholds directly from non-object regions in a single input image instead of object regions from large training data as other existing methods. As a result, we can completely remove the need of training data for cascade threshold learning. Experimental results on two problems of object detection and face detection confirm that our proposed method can obtain the same level of accuracy and speed as state-of-the-art cascade DPM methods but it has the benefit of no threshold training data.