Towards a better understanding of emotion in speech, it is important to understand how emotion changes and when it changes. Recognizing emotions using pre-segmented speech utterances results in a loss in continuity of emotions and does not provide insights into emotion changes. In this paper, we propose an investigation into emotion change detection from the perspective of exchangeability of data points observed sequentially using a martingale framework. Within the framework, a per-frame GMM likelihood based approach is proposed as a measure of strangeness from a particular emotion class. Experimental results on the IEMOCAP database demonstrate that the proposed martingale framework offers significant improvements over the baseline GLR method for detecting emotion changes not only between neutral and emotional speech, but also between positive and negative classes along the arousal and valence emotion dimensions.