The aim of this work is to propose a novel dynamic resource allocation strategy for adaptive Federated Learning (FL), in the context of beyond 5G networks endowed with Reconfigurable Intelligent Surfaces (RISs). Due to time-varying wireless channel conditions, communication resources (e.g., set of transmitting devices, transmit powers, bits), computation parameters (e.g., CPU cycles at devices and at server) and RISs reflectivity must be optimized in each communication round, in order to strike the best trade-off between power, latency, and performance of the FL task. Hinging on Lyapunov stochastic optimization, we devise an online strategy able to dynamically allocate these resources, while controlling learning performance in a fully data-driven fashion. Numerical simulations implement distributed training of deep convolutional neural networks, illustrating the effectiveness of the proposed FL strategy endowed with multiple reconfigurable intelligent surfaces.