In this work, we investigate mapping both natural language food and quantity descriptions to matching USDA database entries. We demonstrate that a convolutional neural network (CNN) model with a softmax layer on top to directly predict the most likely database matches outperforms our previous state-of-the-art approach of learning binary classification and subsequently ranking database entries using similarity scores with the learned embeddings. The softmax model achieves 97.3% top-5 USDA quantity and 91.1% food recall over the full database, compared to only 70.0% quantity and 46.4% food recall with a sigmoid model, where top-5 recall indicates the percentage of test cases in which the correct quantity or food is in the top-5 hits. Evaluated on 9,600 spoken meals over all foods, the softmax model achieves 91.6% top-5 quantity and 80.1% food recall. We also explore jointly learning both mappings with a single CNN to boost quantity mapping, and improve food mapping by reranking the food database entries using the predicted quantity matches.