In recent years, it has become a trend for people to manipulate their own portraits before posting them on a social networking service. However, it is difficult to get a desired portrait after manipulation without sufficient experience or skill. To obtain a simpler and more effective portrait manipulation technique, we consider an automated portrait manipulation method based on five impression words: clear, sweet, elegant, modern, and dynamic. In the first part of this work, we analyzed the relationship between deformed portraits (only facial features) and the five impression words for a large dataset that we collected by crowd-sourcing. In the next part, we leveraged the knowledge from the results of the analysis to develop a ranking network trained on the collected data for estimating the impression associated with a portrait. Our network demonstrates superior performance in impression prediction when compared to other state-of-the-art methods.