Single-pixel camera (SPC) captures encoded projections of the scene in a unique detector such that the number of compressive projections is lower than the size of the image. Traditionally, classification is not performed in the compressive domain because it is necessary to recover the underlying image before to classification. Based on the success of Deep Learning (DL) in classification approaches, this paper proposes to classify images using compressive measurements of SPC. Furthermore, the proposed DL approach designs the binary sensing matrix in the SPC to improve the classification accuracy. In particular, a whole neural network is trained to learn the SPC sensing matrix, in the first layer, and extracts features from the single-pixel compressive measurements. The proposed approach overcomes two approaches of the state-of-the-art in terms of classification accuracy.