The paper proposed a method to realize a speech-to-gesture conversion for communication between normal and speech-impaired people. Keyword spotting was employed to recognize the keywords from input speech signals. At the same time, the three dimensional gesture models of keywords were built by 3D modeling technology according to the "Chinese sign language". The speech-to-gesture conversion was finally realized by playing the corresponding 3D gestures with OpenGL from the results of keyword spotting. Tests show that the realized keyword spotting achieves 90.1% of average recognition rate on letters and numbers. The converted gestures obtain 4.4 of the mean opinion score. Therefore the proposed method can be applied to the communications between normal and speech-impaired people.