Estimating physical activity (PA) intensity and energy expenditure (EE) is a problem that typically requires the use of wearable sensors such as a heart rate monitor, or accelerometer. We investigate the accuracy of a computer vision system using videos recorded from a pair of wearable video glasses to estimate PA strength and EE automatically using age, gender, speed, and activity cues. Age and gender are obtained using the Deep EXpectation network, while activity is estimated from joint angles and movement speed. We also present results on a study of 50 participants performing four different activities while measuring corresponding features of interest such as height, weight, age, sex, and ground truth EE and PA strength data collected via accelerometer. We present both the results of each computer vision subsystem and overall accuracy of the PA strength estimation (89.5%) and the average EE difference (1.96 kCal/min).