The inherent dependencies among multiple physiological signals are crucial for multimodal emotion recognition, but have not been thoroughly exploited yet. This paper propose to use restricted Boltzmann machine (RBM) to model such dependencies.Specifically, the visible nodes of RBM represent EEG and peripheral physiological signals, and thus the connections between visible nodes and hidden nodes capture the intrinsic relations among multiple physiological signals. The RBM also generate new representation from multiple physiological signals. Then, a support vector machine is adopted to recognize users’ emotion states from the generated features. Furthermore, we extend the proposed fusion method for incomplete datas, since physiological signals are often corrupted due to artifacts. Specifically, we pre-train the RBM using all the complete data, then we update missing values and RBM parameters to minimize free energy of visible vectors using both complete and incomplete data. Experiments on benchmark databases demonstrate the effectiveness of the proposed methods.