This paper presents a novel algorithm for fast and effective vanishing point detection. Once line segments in an input image are detected by LSD algorithm, the proposed method filters out outlier line segments. The remaining line segments are then over-clustered, and each cluster is assigned to 5 different types. According to the assigned type, each cluster is re-merged by applying different criteria, and the re-merged clusters generate hypotheses for vanishing points. Vanishing points are finally detected by utilizing these hypotheses and objective function minimization which reflects orthogonality of vanishing points. The proposed method is accurate because the proposed line over-clustering minimizes erroneous clusters, and type assignment is used for precise re-merging. Furthermore, the proposed method is fast since re-merging is conducted on a cluster level and the objective function is minimized non-iteratively. Experimental results show that the proposed method outperforms the state-of-the-art methods in terms of accuracy and computational cost.