Analysis of hand skeleton data can be used to understand patterns in manipulation and assembly tasks. This paper introduces a graphbased representation of hand skeleton data and proposes a method to perform unsupervised temporal segmentation of a sequence of subtasks in order to evaluate the efficiency of an assembly task. We explore the properties of different choices of hand graphs and their spectral decomposition. A comparative performance of these graphs is presented in the context of complex activity segmentation. We show that the spectral graph features extracted from 2D hand motion data outperform the direct use of motion vectors as features. We also make the collected hand position data available to the research community to facilitate further development in this direction.