When using optimization methods with matrix variables in signal processing and machine learning, it is customary to assume some low-rank prior on the targeted solution. Nonnegative matrix factorization of spectrograms is a case in point in audio signal processing. However, this low-rank prior is not straightforwardly related to complex matrices obtained from a short-time Fourier -- or discrete Gabor -- transform (STFT), which is generally defined from and studied based on a modulation operator and a translation operator applied to a so-called window. This paper is a first study of the low-rankness property of time-frequency matrices. We characterize the set of signals with a rank-$r$ (complex) STFT matrix in the case of a unit hop size and frequency step with few assumptions on the transform parameters. We discuss the scope of this result and its implications on low-rank approximations of STFT matrices.