Affect-Sensitive Human-Computer Interaction is enjoying growing attention. Emotions are an essential part of interaction, whether it is between humans or human and machine. This paper analyses the interaction of a user with four different virtual avatars, each manifesting distinct emotional displays, based on the principles of Affect Control Theory. Facial expressions are represented as a vector in a 3D continuous space and different sets of static visual features are evaluated for facial expression recognition. A probabilistic framework is used to simulate the interaction between the user and the virtual avatar. The results demonstrate that the probabilistic framework enables the system to perceive user’s and agent’s feelings.