We give a brief history of the performance analysis of LMS.
Using averaging theory we show when and why the ‘independence
assumption’ ‘works’; we preface this with a fast
heuristic explanation of averaging methods, clarifying their
connection to the ‘ODE’ method. We then extend the discussion
to more recent distributed versions such as diffusion
LMS and consensus. While single node LMS is a single timescale
algorithm it turns out that distributed versions are twotime
scale systems, something that is not yet widely understood.
We sketch open problems such as tracking behaviour,
which despite all the research remains little addressed. We
also make the connection with the parallel literature in adaptive
control. We comment on the recent upsurge of interest in
‘online learning’ in machine learning (which emphasizes decaying
gains). Finally we speculate on future developments
in the three areas with reference to LMS.