Children's exposure to violence has become a severe problem with the rapid development of Internet. Recognizing violent video and estimating violence extent become crucial. Most researches focus on violent scene or violent action detection, lacking overall violence extent information. In this paper, we propose a violence rating prediction approach and build a novel violent video dataset. Our proposed method has two advantages: (1) videos are represented by features extracted from a learned two-stream network; (2) relationship between different violence extent can be learned and utilized to predict violence rating. To demonstrate the effectiveness of our method, we created a well-labelled dataset which contains 1,930 violent videos. Each video is labelled with 6 objective violent attributes. Furthermore, we employ pairwise comparison method to obtain ground-truth violence rating for each video. Our proposed approach was evaluated on our dataset. Its results showed that our proposed method outperforms the state-of-art video classification methods.