Human adapt their behaviors by continuously monitoring one another to function socially in our society. The ability to process face identity from memory is a crucial basic capability. In this work, we propose an event-contrastive connectome network (E-cCN) in representing brain’s functional connectivity with novel contrastive loss to handle layers of fMRI data variabilities exists under different controlled stimuli events to achieve improved automatic assessing of an individual’s face processing and memory ability. Our proposed connectome network achieves an overall recognition accuracy of 80.20% and 82.05% in binary classification of separating high versus low scoring subjects on tasks of Taiwanese Face Memory Test (TFMT) and component inverse efficiency score (cIE) respectively. Further, our network embedding representation demonstrate distinct connectivity patterns in key face processing brain regions (ROIs) when comparing between high and low face processing and memory ability.