Mid Semester 2024 — the paper, worked through.

The question, verbatim

1. Explain how we can enable handling long distance dependencies in RNNs. [5 marks]

How to approach it

Only five marks, so keep it tight. Start with the failure mode a vanilla RNN has — the vanishing / exploding gradient in back-propagation through time — because "enable handling" only makes sense once you have named what stops it. Then give the fix: gating, in the LSTM's cell-state form and the GRU's update-gate form, both of which turn the multiplicative gradient path into a nearly additive one. Close with the more radical answer — attention removes recurrence altogether. Five marks, five distinct claims.

Worked answer

Where the problem comes from

A vanilla RNN reads a sequence one token at a time and updates a hidden state through a shared recurrence:

In principle, this recurrence can carry information from step 1 to step 100 — the hidden state at any late position is a function of every earlier one. But training with back-propagation through time unrolls the network into a $T$-layer deep computation, and the gradient of the loss with respect to an early hidden state is a product of Jacobians, one per time step. That product either vanishes (spectral norm < 1, geometric decay to zero) or explodes (spectral norm > 1, geometric blow-up). Long-range gradients never reach the early weights — so a vanilla RNN has the capacity for long dependencies but not the trainability.

Fix 1 — gating (LSTM, GRU)

Both LSTM and GRU replace the single $\tanh$ update with a small circuit of sigmoid gates that decide, per component, how much of the past to keep and how much of the new candidate to write. On any dimension where the "keep" gate is near 1, the state passes through unchanged and the gradient across that step is close to the identity — the multiplicative gradient path becomes an additive one.

LSTM. Keeps a separate cell state $c_t$ that is updated by an additive rule, gated by a forget gate $f_t$ and an input gate $i_t$:

This is the famous constant error carousel: when the forget gate is near 1, gradients flow along the cell-state highway across many time steps essentially unchanged. Long dependencies stop being killed by BPTT.

GRU. Fuses the cell and hidden state into one vector and uses only two gates — an update gate $z_t$ and a reset gate $r_t$. The update gate interpolates between the old state and the new candidate:

When $z_t$ is near 0, the past is preserved verbatim and the gradient across that step is again close to the identity. Fewer parameters than LSTM, same additive-highway trick.

Fix 2 — get rid of recurrence entirely (attention / self-attention)

The other answer is more radical: if recurrence is the reason long-range signal is fragile, remove it. Self-attention lets any position look directly at any other position with a single matrix multiply — the distance-1 shortcut a plain RNN never had. Every token has an $O\left(1\right)$ path length to every other token, so long-range dependencies stop being a training problem at all. This is the escape hatch the transformer takes, and it is what makes the rest of this paper (Q3, Q4) possible.

The question, verbatim

2. Why is attention needed in a sequence-to-sequence model? Explain with attention function description. [5 marks]

How to approach it

The question is two halves. First, why: name the bottleneck in vanilla seq2seq and argue that different target words need different source content, so the decoder needs a fresh source view at each step. Second, the function: write the three-line attention block — score, softmax, weighted sum — and name at least one concrete score function (additive Bahdanau or multiplicative Luong). Five marks, so keep it to one crisp diagram-in-words plus the equations.

Worked answer

Why attention is needed

A vanilla encoder-decoder for machine translation (Sutskever et al. 2014) reads the whole source sentence with an RNN and summarises it into a single fixed-length vector — the final encoder hidden state — which is then handed to the decoder as its only source-side input. The decoder must generate the entire target sentence from that one vector.

Two problems fall out of this design:

1. The bottleneck. A fixed-size vector cannot carry all of a long source sentence. Empirically, translation quality drops sharply once the source exceeds roughly 20–30 tokens, because the encoder is being asked to compress unbounded content into a bounded state.
2. Different targets need different source content. Consider translating "the red car is fast" into French — "la voiture rouge est rapide". When the decoder is producing voiture, it needs to look at car. When it is producing rouge, it needs to look at red. One fixed vector cannot serve both — it averages them or forgets one. What the decoder actually needs is a different source view at every target step.

Attention (Bahdanau et al. 2015) fixes both. The encoder keeps all its per-position hidden states $h_1,\dots ,h_n$. At each decoder step, a soft alignment is computed over those states and a fresh context vector $c_t$ is produced — a weighted sum of encoder states, weighted by how well each source position matches the current decoder state. The decoder can "look back" at any source token at any time.

The attention function

At decoder step $t$, attention runs three lines:

The score function is the model's judgement of how relevant source position $i$ is to producing the next target token. Two standard choices:

Additive (Bahdanau, 2015) — a tiny one-hidden-layer feedforward network on the concatenated decoder and encoder states:

Multiplicative / dot (Luong, 2015) — a bilinear form, cheaper because it is essentially a matrix multiply:

Once $c_t$ is computed, the decoder produces the next word from a combination of its own state and the context vector: $p\left(y_t\right|y_{<t},x)=\mathrm{softmax}(W[s_t;c_t])$.

The question, verbatim

3. Describe the transformer encoder block in detail. [10 marks]

How to approach it

A ten-mark structural question. Walk from input to output, naming each sub-component and giving its defining equation. The full checklist: input embedding + positional encoding, multi-head self-attention (with the scaled dot-product formula and how heads are computed and combined), residual connection and LayerNorm, position-wise feedforward network, residual and LayerNorm again. Close with the complexity $O(n^2·d)$ and one line on why the block looks this way. Aim for one equation-plus-paragraph per sub-component — that is roughly two marks each, which lines up with the ten.

Worked answer

Setup — the block's input and output

An encoder block takes a matrix $x\in {ℝ}^{n\times d_{\mathrm{model}}}$ — one row per input token, embedded into a shared $d_{\mathrm{model}}$-dimensional space. It returns a tensor of the same shape, so blocks stack cleanly. A transformer encoder is $N$ such blocks in series (the original paper uses $N=6$).

Before the first block, two things happen once to the input:

1. Each token id is turned into a $d_{\mathrm{model}}$-dimensional token embedding via a learned lookup table.
2. A positional encoding — a fixed sinusoidal signal or a learned vector — is added to each embedding. Self-attention is otherwise position-agnostic; without this, the model would treat the sentence as a bag of tokens.

Sub-layer 1 — multi-head self-attention

The first sub-layer lets each token look at every other token in the sequence. For a single head, three learned projections turn the input into queries, keys, and values:

where $W_Q,W_K\in {ℝ}^{d_{\mathrm{model}}\times d_k}$ and $W_V\in {ℝ}^{d_{\mathrm{model}}\times d_v}$. Scaled dot-product attention is then:

Reading it row by row: for each query token, dot it with every key to get a raw score, divide by $\sqrt{d_k}$ so the softmax does not saturate at high dimension, softmax over the sequence to get attention weights, and take a weighted sum of the value vectors. Each output row is a soft lookup — a mixture of value vectors, weighted by query-key similarity.

Multi-head. Instead of one large attention with dimension $d_{\mathrm{model}}$, the block runs $h$ heads in parallel with per-head dimension $d_k=d_{\mathrm{model}}/h$. Each head has its own $W_Q,W_K,W_V$. Their outputs are concatenated and projected back:

Different heads learn to attend to different kinds of structure — one may track syntactic dependencies, another may track co-reference. Splitting into heads is essentially free (the total parameter count is roughly the same as a single-head attention of dimension $d_{\mathrm{model}}$) but is strictly more expressive.

Residual connection and LayerNorm (round 1)

The attention output is added back to its input via a residual connection, and the sum is normalised:

The residual gives gradients a direct path around the sub-layer, letting deep stacks train stably. LayerNorm normalises each token vector to zero mean and unit variance across its features, then rescales with learned per-feature $\gamma$ and $\beta$. Unlike BatchNorm, it depends only on the token itself, so it works with variable sequence lengths and small batches.

Sub-layer 2 — position-wise feedforward network

The second sub-layer is a small MLP applied independently to each position. Two linear layers with a non-linearity in between:

with $W_1\in {ℝ}^{d_{\mathrm{model}}\times d_{\mathrm{ff}}}$ and $W_2\in {ℝ}^{d_{\mathrm{ff}}\times d_{\mathrm{model}}}$ — the inner dimension is typically $4d_{\mathrm{model}}$. The original paper uses ReLU; modern implementations (BERT, GPT-2 onwards) use GELU. The point is that self-attention only mixes information across positions linearly; the FFN adds a per-token non-linear transformation, and does most of the block's per-position compute.

Residual and LayerNorm (round 2)

Same pattern as after attention:

$z_2$ is the output of one encoder block, ready to be fed into the next.

The block, at a glance

Step	What it does	Shape
Input	token embedding + positional encoding	$n\times d_{\mathrm{model}}$
Multi-head self-attention	each token attends to every token; $h$ heads run in parallel	$n\times d_{\mathrm{model}}$
Add & Norm	residual, then LayerNorm across features	$n\times d_{\mathrm{model}}$
Position-wise FFN	two-layer MLP, applied independently at each position	$n\times d_{\mathrm{model}}$
Add & Norm	residual, then LayerNorm again	$n\times d_{\mathrm{model}}$

Complexity

Self-attention forms an $n\times n$ attention matrix, so the block is $O(n^2·d_{\mathrm{model}})$ in time and $O\left(n^2\right)$ in memory. Quadratic in the sequence length. This is the cost the field has spent the last five years trying to reduce — Flash Attention, sparse attention, linear attention, state-space models. But inside one encoder block, this cost is what buys you an $O\left(1\right)$ path length between any two tokens, which is the whole point.

The question, verbatim

4. How does LoRA Finetuning approach differ from Adapters? Explain with architectures. [10 marks]

How to approach it

Frame both as parameter-efficient fine-tuning (PEFT) — same problem (fine-tune a large pretrained model cheaply by touching only a small number of new parameters, keeping the original weights frozen), two different architectural answers. Sketch each one: where it plugs into the transformer block, what its forward equation looks like, how many parameters it adds. Then give the differences on axes that actually matter: serial vs parallel placement, inference latency, mergeability, expressiveness. The word "architectures" in the question is a rubric signal — you need diagrams-in-words or pictures, not just prose.

Worked answer

The shared problem

Full fine-tuning of a modern pretrained transformer means updating all of its billions of parameters and storing a full-sized checkpoint per downstream task. That is expensive to train and impossible to serve if you have many tasks. Parameter-efficient fine-tuning methods freeze the original weights $W$ and add a small number of new, trainable parameters — typically 0.1%–5% of the base model — that specialise it to the task. Two of the most widely used are Adapters (Houlsby et al. 2019) and LoRA (Hu et al. 2021).

Adapters — architecture

An adapter is a small bottleneck MLP inserted inside the transformer block, in series with the frozen sub-layers. Given an input $h\in {ℝ}^d$, the adapter does:

Three pieces: a down-projection $W_{\mathrm{down}}\in {ℝ}^{r\times d}$ that squashes to a small bottleneck of size $r\ll d$, a non-linearity $\sigma$ (GELU / ReLU), and an up-projection $W_{\mathrm{up}}\in {ℝ}^{d\times r}$ back to the model dimension. A residual connection carries the input around it, and $W_{\mathrm{up}}$ is initialised to zero so the adapter starts as an identity — the model behaves exactly like the frozen base at initialisation and only drifts as the adapter learns.

The adapter is inserted twice per transformer block: once after the multi-head self-attention sub-layer, once after the FFN sub-layer. The block becomes:

x
 ├─> MultiHead-SelfAttn (frozen)
 ├─> Add + LayerNorm
 ├─> Adapter  (trainable, ~few % params)
 ├─> FFN (frozen)
 ├─> Add + LayerNorm
 ├─> Adapter  (trainable)
 └─> out

Only $W_{\mathrm{down}}$, $W_{\mathrm{up}}$, and the LayerNorm parameters are trained. Everything else is frozen. Adapters typically add ~1–5% of the base model's parameter count per task.

LoRA — architecture

LoRA does not insert anything into the block. It adds a trainable low-rank update in parallel to selected frozen weight matrices — typically the query and value projections inside self-attention. If $W\in {ℝ}^{d\times d}$ is one of the frozen projections, LoRA replaces it with:

with $A\in {ℝ}^{r\times d}$ and $B\in {ℝ}^{d\times r}$, $r\ll d$. The factor $\alpha /r$ is a scaling constant. $A$ is initialised at random and $B$ is initialised to zero, so at the start of training $BA=0$ and the model is exactly the frozen base — the same "start as identity" trick as adapters.

Forward pass through a LoRA-augmented projection:

Two matrix multiplies down the low-rank path, in parallel with the frozen $Wx$. Trained parameters per LoRA-adapted matrix: $2dr$. With rank $r=8$ or 16 applied to just $W_Q$ and $W_V$, LoRA typically trains <1% of the base model's parameters.

x ──> W (frozen)     ──┐
      │                 ├─> add ─> h
      └─> A ─> B (train)┘

The critical differences

Axis	Adapters	LoRA
Where it sits	Inside the block, after each sub-layer	Parallel to chosen weight matrices (usually $W_Q$, $W_V$)
Placement in the graph	Serial — the forward pass goes through it	Parallel — added to the frozen matmul
Inference latency	Adds latency at every layer, both training and inference	Zero extra latency at inference — the update $BA$ can be merged into $W$: $W^{\prime }=W+BA$
Trained shape	Bottleneck MLP with a non-linearity	Linear low-rank factorisation, no non-linearity
Trainable params	~1–5% of the base	Often <1% of the base (tunable via $r$)
Task switching	Swap adapter weights in place	Swap by unmerging one $BA$ and merging another (or serve unmerged)

Why each design exists

Adapters came first and are the more general recipe — the bottleneck can learn any non-linear transformation, and the same trick generalises to inserting small modules anywhere in the graph. Their weakness is that they add serial depth: every adapter is one more layer of forward computation the request has to pass through, and stacking them across all 24 or 32 transformer blocks measurably slows inference.

LoRA takes a specific bet: the update $\Delta W$ that fine-tuning would learn is approximately low-rank, so it can be represented as $BA$ with very small $r$. In exchange for that assumption, LoRA gets three practical wins — fewer trained parameters, no inference latency (thanks to mergeability), and effortless multi-task serving (keep $W$, keep a stack of small $(A,B)$ pairs on disk, load the pair for the task you want).