In this paper, the Multi-view Variational Recurrent Neural Network (MvVRNN) is proposed for multi-modal human emotion recognition with gaze and brain activity data while humans view images. For realizing accurate emotion recognition, we focus on the following three characteristics of biological signals: 1) the relationship between implicit and explicit information such as gaze and brain activity data, 2) the temporal changes related to human emotions and 3) the effects of noises that can be included during data acquisition. For treating these characteristics, the proposed MvVRNN has several mechanisms including 1) the integration of multi-modal information including implicit and explicit states of humans, 2) the recurrent module for sequential data and 3) the variational approximation based on the Gaussian distribution. The experimental results show that emotion recognition based on the MvVRNN outperforms several existing methods.