Solar energy adoption is moving at a rapid pace. The variability in solar energy production causes grid stability issues and hinders mass adoption. To solve these issues, more accurate photovoltaic power forecasting systems are needed. In intra-hour forecasting, the most challenging issue is high output fluctuations due to cloud motion, which can occlude the sun. Using ground-based sky images, this paper proposes two convolutional neural network models for intra-hour nowcasting and forecasting that incorporate physical information on sun motion and cloud coverage by means of the sun area mean pixel intensity. Particularly, our models exploit that information instead of relying exclusively on photovoltaic output history data as it is standard in state of the art. Taking advantage of sun position and cloud coverage information, we were able to reduce the overall root mean squared error for the nowcasting task, making the model more accurate especially during cloudy days, and obtaining competitive results on forecasting. Moreover, our models are more robust against artifacts such as occlusion and noisy observations.