Linear RNNs.
A student-led seminar on recurrent models with linear state updates — the family that ties together linear attention, state-space models (Mamba, S4), and modern efficient sequence architectures. This page will hold the slide-by-slide study companion once the deck is available.
§0Status
The seminar hasn't been presented yet, so no slides are available. Once the deck lands, this page will get the same slide-by-slide walkthrough treatment as the other seminars — every slide embedded verbatim with a first-principles explanation underneath.
§1What the seminar covers
Linear RNNs are recurrent networks whose state update is a linear function of the previous state and the current input: something of the form . Removing the nonlinearity from the recurrence buys a lot: the hidden state can be computed in parallel across time (through associative scans or FFT-like reductions), so training doesn't sequentially bottleneck on the recurrence.
This is the algebraic backbone behind the recent efficient-sequence-model line: linear attention, State-Space Models (S4, S5, Mamba), and the various gated linear architectures. The seminar likely covers the recurrence formulation, why it can be parallelised, how it relates to attention, and which variants have caught on in practice.
Linear attention — the closest cousin covered in a main lecture — appears in Lecture 6 §5. That section derives the kernel-based rewrite, the causal-case recurrence, and the "transformers are RNNs" framing that this seminar builds on.