Being affected by mental stress during conversations might have a direct or indirect effect on our speech acoustics as well as on our physiological responses. This paper presents a study on finding the relationship between these two modalities, speech acoustics and physiology, during stressful conversations between humans. Heart rate and respiratory sinus arrhythmia have been considered as physiological variables in the present study. Two datasets, one from stress induction sessions and the other one from in-lab discussions of relationship conflicts between couples, have been analyzed. A series of experiments have been performed separately on the two datasets, as well as on the combined dataset. The research finds acoustic features that are significantly correlated with the physiological variables during stressful conversations. It also predicts the physiological signals from speech features through a nonlinear regression analysis. The results take us one step forward towards building an extremely non-intrusive and relatively inexpensive method of predicting physiological responses from speech, and thus detecting the presence and quantifying the intensity of stress during stressful conversations.