Attention networks constitute the state-of-the-art paradigm for capturing long temporal dynamics. This paper examines the efficacy of this paradigm in the challenging task of emotion recognition in dyadic conversations. In this work, we introduce a novel attention mechanism capable of inferring the immensity of the effect of each past utterance on the current speaker emotional state. The proposed self-attention network captures the correlation patterns among consecutive encoder network states, thus enabling the robust and effective modeling of temporal dynamics over arbitrary long temporal horizons. We exhibit the effectiveness of our approach considering the challenging IEMOCAP benchmark. We show that, our devised methodology outperforms state-of-the-art alternatives and commonly used approaches, giving rise to promising new research directions in the context of Online Social Network (OSN) analysis tasks.