Hand gesture, a common non-verbal language, is being studied for Human Computer Interaction. Hand gestures can be categorized as static hand gestures and dynamic hand gestures. In recent years, effective approaches have been applied to hand gesture recognition. However, almost all of the previous works only focus on either of the two categories instead of both, and none of them has used the temporal information on the recognition of static hand gestures.In this paper, we propose a three-level scheme to utilize the temporal interframe pattern on the recognition of both static and dynamic hand gestures. The first classifier assigns a class label to each frame of the video sequence that contains both static and dynamic hand gestures. The second classifier uses the temporal pattern of the class labels to correct the errors of the first classifier and to distinguish between static and dynamic hand gestures. The third classifier is then used to recognize dynamic hand gestures. We believe that we are the first to propose such a strategy. The extensive experiments showed promising performance and demonstrated the feasibility of using temporal interframe pattern to recognize dynamic hand gestures and to correct the errors in static hand gesture recognition.