In this paper, we consider 2-class and 3-class classification problemsfor classifying patients with Amyotropic Lateral Sclerosis (ALS),Parkinson’s Disease (PD) and Healthy Controls (HC) using a CNN-LSTM network. Classification performance is examined for threedifferent tasks, namely, Spontaneous speech (SPON), Diadochoki-netic rate (DIDK) and Sustained Phonation (PHON). Experimentsare conducted using speech data recorded from 60 ALS, 60 PD and60 HC subjects. Classification using SVM and DNN are consid-ered baseline schemes. Classification accuracy of ALS and HC (in-dicated by ALS/HC) using CNN-LSTM is found to be 89.20% forthe PHON task as compared to 78.80% using the best of the base-line schemes. Furthermore, the CNN-LSTM network achieves thehighest PD/HC classification accuracy of 88.5% for SPON task andthe highest 3-class (ALS/PD/HC) classification accuracy of 85.24%for DIDK task. Experiments using transfer learning at low resourcetraining data show that data from ALS benefits PD/HC classifica-tion and vice-versa. Experiments with fine-tuning weights of 3-class (ALS/PD/HC) classifier for 2-class classification (PD/HC orALS/HC) gives an absolute improvement of 2% classification accu-racy in SPON task when compared with randomly initialized 2-classclassifier.