Rate distortion optimization (RDO) is the basis for algorithm optimization in video coding, such as mode decision, rate control and etc. Minimizing the rate distortion coding cost is usually employed to determine the optimal coding parameters such as quantization level, coding mode, and etc. However, rate and distortion calculations for optimal solution decision from massive possible candidates suffer from dramatically high computation complexity. To resolve this problem, this paper proposes a fast TU level rate model with higher accuracy by fully imitating the behavior pattern hid in entropy. We evaluate the contribution percentages of different syntax elements in entropy coding, and individually develop syntax element-wise accurate rate models to construct the whole TU level model. In addition, coefficient levels are weighted adaptively to distinguish the nonuniform contributions of different coefficients in terms of rate profiling. Moreover, position-wise parameter is defined to depict the distribution patterns for possible non-zero coefficients within one block. The final linear rate model is developed by fine-tuning the model parameters from great amounts of samples in a statistical way. Finally, the transform domain distortion model is also established to bypass the inverse quantization and inverse transform. Experimental results show that the proposed algorithm can achieve 52.68% complexity reduction with 1.67% BD-BR increase for LD configuration, and 49.76% complexity reduction with 1.74% BD-BR increase for RA configuration, respectively.