Epilepsy affects approximately 1% of the world’s population. Semiology of epileptic seizures contain major clinical signs to classify epilepsy syndromes currently evaluated by epileptologists by simple visual inspection of video. There is a necessity to create automatic and semiautomatic methods for seizure detection and classification to better support patient monitoring management and diagnostic decisions. One of the current promising approaches are the marker-less computer-vision techniques. In this paper an end-to-end deep learning approach is proposed for binary classification of Frontal vs. Temporal Lobe Epilepsies based solely on seizure videos. The system utilizes infrared (IR) videos of the seizures as it is used 24/7 in hospitals’ epilepsy monitoring units. The architecture employs transfer learning from large object detection "static" and human action recognition "dynamic" datasets such as ImageNet and Kinectics-400, to extract and classify the clinically known spatiotemporal features of seizures. The developed classification architecture achieves a 5-fold cross-validation f1-score of 0.844±0.042. This architecture has the potential to support physicians with diagnostic decisions and might be applied for online applications in epilepsy monitoring units. Furthermore, it may be jointly used in the near future with synchronized scene depth 3D information and EEG from the seizures.