Robust Principal Component Analysis (RPCA) is an optimization problem that decomposes a data matrix into a low-rank and a sparse matrix. However, solving this problem using alternating procedures requires sequentially computing singular value decompositions (SVDs) of large matrices, which is computationally expensive. In this work, we propose a computation protocol that leverages Gauss-Newton iterations to speed up the sequential computation of SVDs and accelerate the entire RPCA process. Our method is validated on synthetic and video data, benchmarked against established RPCA algorithms, and analyzed for stability with respect to hyperparameters. Our proposed protocol can also be applied to problems that require repeated computation of the proximal of functions that solely depend on singular values of the input matrix.