Contextual information such as the co-occurrence of objects and the location of objects has played an important role in object detec- tion. We present candidate pruning and object rescoring methods that leverage contextual information and that can improve the state- of-the-art CNN-based object detection methods such as Fast R-CNN and Faster R-CNN. In our pruning method, we formulate candidate reduction as a Markov random field optimization problem. In our rescoring method, we employ a machine learning technique to recon- sider the detection scores of candidate windows. We experimentally demonstrate improvements in R-CNN-based object detection meth- ods using two datasets. Moreover, we apply our model to the struc- tured retrieval task to show the potential applications of our model.