Automatic facial stroke and palsy assessment systems based on computer vision have strong benefits in their application. We propose a framework that exploits facial graphs with temporal connections and analyses them through a graph attention-based model. The temporal facial graph captures structural and motion cues from the facial image sequence, while the graph attention mechanism effectively analyses the interrelation between facial regions in close proximity. We test the method on one dataset for facial stroke and one for facial palsy and compare the performance with select state-of-the-art methods that employ convolutional neural networks. The experiment results indicate firm improvements associated with the proposed model, with significant improvement in performance over the state-of-the-art in assessing the dataset on facial stroke.