Mid Semester 2023 — the paper, worked through.

The question, verbatim

1. Explain how RNNs enable handling long distance dependencies. Can you compare LSTMs and GRUs at an architectural level? [5 + 5 marks]

How to approach it

The two sub-parts want different things. Sub-part (a) is a mechanism story: what does an RNN do that lets information from step 1 reach step 100. This means writing the recurrence, then honestly explaining the failure mode (vanishing gradients) and how gating is the general fix. Sub-part (b) is a side-by-side spec sheet: same structure of "here are the gates, here is the state, here are the equations", once for LSTM and once for GRU. The key's "only arch" note tells you to skip discussion of accuracy, dataset size, or training time; just architecture.

Worked answer

(a) How RNNs enable long-distance dependencies · 5 marks

A vanilla RNN reads a sequence $x_1$, $x_2$, …, $x_T$ one token at a time and maintains a hidden state $h_t$ that is recurrent: the state at step $t$ is a function of the previous state and the current input:

This is the mechanism that enables long-distance dependencies in principle. Because $h_t$ is a function of $h_{t-1}$, and that of $h_{t-2}$, the hidden state at time 100 has a functional path back to the input at time 1. Information about earlier tokens is carried forward through this shared recurrence, and the same weight matrices are reused at every step, so the model can learn a rule like "if a plural subject appeared, the verb far away must agree" without needing a separate parameter per gap length.

Training uses back-propagation through time (BPTT): the network is unrolled into a $T$-layer feed-forward net that shares weights across layers, and the gradient of the loss with respect to $h_t$ flows backward through every previous step. Every backward step multiplies by the Jacobian $\frac{\partial h_t}{\partial h_{t-1}}$, which is bounded by the spectral norm of $W_{hh}$ times the derivative of $\tanh$.

This is where the vanishing / exploding gradient problem enters. If that Jacobian has spectral norm less than 1, gradients decay geometrically across steps and long-range signal never reaches the earlier weights; if greater than 1, they explode. So the architectural capacity for long dependencies (a recurrence) is real, but a plain RNN fails to train them in practice.

The fix that makes long dependencies actually learnable is gating. Instead of a single non-linear squash, a gated cell introduces sigmoid gates that decide, per component, how much of the old state to keep and how much of the new input to write. When a gate is near 1 on the "keep" path, the gradient flows across that time step nearly unchanged. This is the additive skip connection through time that gives LSTMs and GRUs their long-range memory.

(b) LSTM vs GRU · architectural comparison · 5 marks

Both are gated RNN cells that replace the single $\tanh$ update above with a small circuit of gates. They differ in how much state they carry and how many gates they use to route it.

LSTM. Keeps two state vectors: a cell state $c_t$ (long-term memory) and a hidden state $h_t$ (short-term / output). Three sigmoid gates control it: forget $f_t$, input $i_t$, output $o_t$.

The forget gate erases parts of the old cell state; the input gate writes a candidate; the output gate decides what of the cell state becomes the exposed hidden state. The cell-state line is the constant error carousel: on any dimension where $f_t$ is near 1, the state passes through unchanged and gradients flow back undamped.

GRU. Collapses the cell state and hidden state into a single vector $h_t$, and uses only two gates: update $z_t$ and reset $r_t$.

The reset gate decides how much of the past to look at when computing the candidate. The update gate is a single interpolation knob between "keep the old state" and "replace with the new candidate", so it plays the role of the LSTM's forget and input gates fused into one. There is no separate output gate; the hidden state is exposed directly.

The side-by-side

	LSTM	GRU
State vectors	$c_t$ and $h_t$	only $h_t$
Gates	3 · forget, input, output	2 · update, reset
How "keep" is expressed	independent $f_t$ and $i_t$	tied via $z_t$ and $1-z_t$
Exposed state	gated: $o_t\odot \tanh \left(c_t\right)$	same as internal $h_t$
Parameter count (hidden size $d$)	≈ 4 · ($d$$+$$d$) · $d$	≈ 3 · ($d$$+$$d$) · $d$
Constant-error path	through $c_t$ when $f_t$≈1	through $h_t$ when $z_t$≈0

Similarities. Both are gated recurrences, both use sigmoid gates and a $\tanh$ candidate, both preserve gradient flow through an additive path over time, and in practice both achieve comparable accuracy on most language tasks.

What the key says

The key's four bullets (recurrence, backprop, nature of gates, similarity) line up with the solution above. The top-margin table (GRU 2 gates / no external memory / less params / less memory / faster vs LSTM 3 gates / computationally faster) is the marker's rubric for the comparison sub-part, and the side-by-side above hits every one of those rows. The vertical "only arch" reminder is the trap: do not lose time discussing which one wins on which dataset. The question is architectural only.

The question, verbatim

2. Describe the ELMO Contextual embedding model. How is it used for downstream tasks? [5 + 5 marks]

How to approach it

Sub-part (a) wants the architecture: a stacked bidirectional LSTM language model, trained on a next-token prediction objective in both directions, whose per-layer hidden states become word representations. Sub-part (b) wants the recipe: one embedding per token is a learned weighted sum of the layer states, where the weights are task-specific parameters that are trained together with a downstream head while the ELMo LM itself is frozen. The key's four bullets are exactly this: stacked BiLSTM, weighted sum of layer embeddings, weights are trained, plug into downstream task.

Worked answer

(a) What ELMo is · 5 marks

ELMo (Embeddings from Language Models, Peters et al. 2018) is a contextual word-representation model: unlike GloVe or word2vec, the vector it assigns to the word bank depends on the sentence it appears in. The vector for bank in "river bank" differs from the vector for bank in "savings bank".

Architecturally, it is a stacked bidirectional LSTM language model trained on a large monolingual corpus:

1. A character-CNN encodes each input token into a fixed vector $x_k$. This makes ELMo robust to out-of-vocabulary words, so it can embed any string.
2. A forward LM is a stack of $L$ LSTM layers (typically $L=2$) trained to predict $x_{k+1}$ from $x_1,\dots ,x_k$.
3. A backward LM of the same shape is trained to predict $x_{k-1}$ from the right context.
4. The two directions are trained jointly, maximising the sum of the forward and backward log-likelihoods.

After training, for a token at position $k$ the model produces $2L+1$ representations: the token embedding $x_k$, plus, at every layer $j$, the concatenation of the forward and backward hidden states:

These per-layer vectors are the raw output of ELMo. The empirical observation Peters et al. make is that different layers capture different linguistic information: the lower layer is closer to syntax (POS, morphology), the higher layer is closer to semantics (word sense, coreference). That is why the final embedding is not just the top layer but a mixture across layers.

(b) How it is used for downstream tasks · 5 marks

For a downstream task, say NER or sentiment, ELMo is not fine-tuned. It is used as a frozen feature extractor. For each token, the task model receives one vector produced by taking a learned weighted sum of the ELMo layers, scaled by a per-task gamma:

Here $s_j^{\mathrm{task}}$ are softmax-normalised layer weights (they sum to 1) and ${\gamma }^{\mathrm{task}}$ is a scalar that scales the mixed vector to a magnitude the downstream head is comfortable with. Both are learned during downstream training; they are the only ELMo-side parameters the task ever updates.

The full recipe:

1. Take the pre-trained ELMo LM and freeze it.
2. Run the input sentence through it once to get $h_{k,j}^{\mathrm{LM}}$ for every token and every layer.
3. Compute ${\mathrm{ELMo}}_k^{\mathrm{task}}$ with the current learned $s_j$ and $\gamma$.
4. Concatenate that vector with a static input embedding (GloVe / word2vec) if one is used, giving $[x_k;{\mathrm{ELMo}}_k^{\mathrm{task}}]$.
5. Feed that concatenated vector into the task-specific model (a BiLSTM for NER, a classifier for sentiment, an encoder for QA) and train that model end-to-end. Gradients flow back into $s_j$, $\gamma$, and the task model, not into the ELMo LSTMs.

Optionally, ELMo can also be inserted at the output side of the task model instead of the input, and the paper shows that using it at both ends helps some tasks.

What the key says

The key's four bullets (stacked Bi LSTM, weighted summation of layer embeddings, weights to be computed, downstream task) are exactly the four things above. "Weights to be computed" specifically means the $s_j$ and $\gamma$ are learned, not fixed. A common mistake is to describe ELMo as "just concatenate all the layers" or "just take the top layer", both of which drop that mark.

The question, verbatim

3. Describe a sequence-to-sequence machine translation model where decoder pays attention to the encoder. Why is the attention needed? [3 + 7 marks]

How to approach it

The mark split is the most important thing on this question. The 3-mark half is describing an encoder-decoder RNN with attention: a diagram plus one paragraph is enough. The 7-mark half is why attention is needed. Do not answer this with a single sentence. The full argument has three moves: the bottleneck problem in vanilla seq2seq, the "different target words need different source content" observation from the key, and the concrete alignment computation that solves it.

Worked answer

(a) The model · 3 marks

A sequence-to-sequence machine translation model with attention has three pieces:

1. An encoder BiLSTM reads the source sentence $x_1,\dots ,x_n$ and produces one hidden state per source position, $h_1,\dots ,h_n$. These are the encoder annotations.
2. A decoder LSTM generates the target sentence $y_1,\dots ,y_m$ one token at a time. At target step $t$, its hidden state is $s_t$.
3. An attention block sits between the encoder and the decoder. At every decoder step, it looks over all encoder annotations and produces a context vector $c_t$: a weighted sum of the encoder states, where the weights are computed from how well each source state matches the current decoder state.

The decoder then produces the next target token from its own state and this context vector: $p\left(y_t\right|y_{<t},x)=\mathrm{softmax}(W[s_t;c_t])$.

(b) The attention function · part of 7 marks

At decoder step $t$, attention runs three lines:

The score function is the "attention func" the key wants defined. Two standard choices:

Additive (Bahdanau, 2015). A tiny feed-forward net:

Multiplicative / dot (Luong, 2015). Cheaper: just a dot product, optionally with a learned matrix in the middle:

Once $c_t$ is computed, the decoder uses both its own state and the context to predict the next word. In Bahdanau's formulation the context is fed into the decoder RNN update; in Luong's it is concatenated with the decoder state after the update. Either way, the decoder step becomes conditioned on a source view that changes at every target position.

(c) Why attention is needed · the rest of 7 marks

The bottleneck problem. A vanilla seq2seq encoder-decoder (Sutskever et al. 2014) reads the whole source sentence and summarises it into a single fixed-length vector (the final encoder hidden state) and hands that one vector to the decoder as its only source-side input. The decoder must generate the entire target sentence from this single vector. Empirically this collapses on long sentences: translation quality drops sharply once the source is longer than roughly 20 to 30 tokens, because a fixed-size vector cannot carry all of the source content.

Different targets need different source content. This is the key's core observation. Consider translating "the red car is fast" into French, "la voiture rouge est rapide". When the decoder is producing voiture, it needs to look at car. When it is producing rouge, it needs to look at red. A single fixed source vector cannot serve both; it either averages the two signals or forgets one. What the decoder actually needs is a different source view at each target step. That is exactly what attention gives: a fresh $c_t$ per step, computed by softly pointing at the source tokens that matter for the current output.

Attention lifts the bottleneck by letting the decoder look back. The encoder no longer has to fit the whole source into one vector, because every one of its per-position states $h_i$ is preserved and reachable. At each decoder step, the softmax over alignment scores acts as a soft pointer: it selects (or blends) the source positions relevant to what the decoder is about to output. In effect, the decoder can "look back" at any encoder state at any time, cost-free at inference.

The alignment side-effect. The attention weights ${\alpha }_{t,i}$ also give a free, interpretable soft alignment between source and target words, which was previously a separate step in phrase-based statistical MT systems. Attention absorbs alignment into the model.

What the key says

The key's two bullets are (i) describe a RNN/LSTM with attention "func" defined, hit in the alignment score above, and (ii) different target words need different content words for translation, hit in the "why" argument. The mark split (3 + 7) is worth remembering next year: describing the model is cheap, defending attention is where the marks live.