Neural spiking responses are generated by both extrinsic covariates such as sensory variables and intrinsic covariates such as those rep-resenting the state of a system. Although the external covariates can be directly controlled or measured; the internal factors are hard, if not impossible, to control or even observe. This study provides a statistical framework that enables characterization of the unobserved factors controlling neuronal response variability induced by behavior, with the model parameters fitted directly to real spiking data. We apply this model to simultaneously recorded spiking and local field potential activities of visual cortical neurons, and show its ability to encode and decode time-varying visual information, as well as to capture the dynamic response characteristics underpinning those computations.