A human emotion recognition using multi-modal biological signals based on time lag-considered correlation maximization is presented in this paper. Various multi-modal emotion recognition methods for visual stimuli have been studied and they focus on gaze and brain activity data. The visual stimuli captured by human eyes are sent to the brain by neurotransmitters. Thus, there is a time lag between gaze data, which record where humans gaze at, and brain activity data. However, most of the previous methods only integrate features obtained from each data without considering such a time lag. The proposed method newly introduces the mechanism to consider the time lag into the canonical correlation analysis scheme by assuming that the influence of the visual stimuli on brain activity data follows the Poisson distribution. The contribution of this paper is the construction of a recognition method with considering the time lag for getting truly close to the realization of the occurrence mechanism of human emotions. Experimental results show the effectiveness of considering the time lag between gaze and brain activity data.