In this paper, we develop a framework to achieve a desirable trade-off between fairness, inference accuracy and privacy protection in the inference as service scenario. Instead of sending raw data to the cloud, we conduct a random mapping of the data, which will increase privacy protection and mitigate bias but reduce inference accuracy. To properly address the trade-off, we formulate an optimization problem to find the optimal transformation map. As the problem is nonconvex in general, we develop an iterative algorithm to find the desired map. Numerical examples show that the proposed method has better performance than gradient ascent in the convergence speed, solution quality and algorithm stability.