This paper presents a novel method to track the hierarchical structure of Web video groups on the basis of salient keyword matching including semantic broadness estimation. To the best of our knowledge, this paper is the first work to perform extraction and tracking of the hierarchical structure simultaneously. Specifically, the proposed method first extracts the hierarchical structure of Web video groups and salient keywords of them on the basis of an improved scheme of our previously reported method. Moreover, to calculate similarities between Web video groups obtained in different time stamps, salient keyword matching is newly developed by considering both co-occurrences of the salient keywords and semantic broadness of each Web video group. Consequently, tracking of the hierarchical structure over time becomes feasible to easily understand popularity trends of many Web videos for realizing effective retrieval.