De novo generation of molecules is a crucial task in drug discovery. The blossom of deep learning-based generative models, especially diffusion models, has brought forth promising advancements in de novo drug design by finding optimal molecules in a directed manner. However, due to the complexity of chemical space, existing approaches can only generate extremely small molecules. In this study, we propose a Graph Latent Diffusion Model (GLDM) that operates a diffusion model in the latent space modeled by a pretrained autoencoder. Applying diffusion processs on latent representations rather than original molecular graphs, GLDM improves training efficiency and enables generation of larger drug-like molecules. GLDM achieves state-of-the-art results on GuacaMol benchmarks.