Compressive spectral imaging (CSI) acquires random projections of a spectral scene. Typically, before applying any post-processing task, e.g. clustering, it is required a computationally expensive reconstruction of the underlying 3D scene. Therefore, several works focus on improving the reconstruction quality by adaptively designing the sensing matrix aiming at better post-processing results. Instead, this paper proposes a hierarchical adaptive approach to design a sensing matrix of the single-pixel camera, such that pixel clustering can be performed in the compressed domain. Specifically, in each step of the hierarchical model, a sensing matrix is designed such that clustering features can be extracted directly from the compressed measurements. Finally, the complete segmentation map is obtained with the majority voting method in the partial clustering results at each hierarchy step. In general, an overall accuracy of 78.94%, and 65.35% was obtained using the ``Salinas'', and `` Pavia University'' spectral image datasets, respectively.