An attention lab notebook.
A practical companion for Advanced NLP. The original slides and past papers stay intact; the notes underneath unpack the parts that usually need a second pass.
§1What this is
Advanced NLP covers a lot of ground quickly: linear algebra, tokenizers, transformer internals, and efficient attention, with past papers in the mix. It is easy to lose the thread when every new idea arrives wrapped in a dense slide.
Think of this as the set of notes you would want beside the deck. Each slide is followed by an explanation of what it is saying, how the pieces connect, and what is worth remembering. The aim is understanding, not a prettier version of the same bullet points.
Start with any lecture and read straight through, or use the section links to jump to a topic. Each Polaroid holds one slide; click it for a larger view. The notes point out the key idea, likely confusion, and any math worth slowing down for.
§2The material
Six lectures, two tutorials, and three past exam papers. The lectures and tutorials are from Monsoon '26; PYQs come from the community iiitprevpapers repository.
Lectures
Logistics
Course structure, grading, the reading list, and the shape of the semester ahead.
Transformers
Why transformers replaced RNNs, then the encoder-decoder block, residual paths, layer norm, and training.
Tokenization
BPE, WordPiece, SentencePiece, and Unigram: how text becomes model input, and why that decision has consequences.
Positional Encoding
Sinusoidal, learned, RoPE, and ALiBi encodings for the deceptively hard job of representing order.
Attention
Queries, keys, values, softmax, multi-head attention, masking, complexity, and worked matrix shapes.
More Attention
Flash, sparse, and linear attention: ways to manage the $O(n^2)$ cost without giving up too much quality.
Normalization
Why normalization matters, then BatchNorm/LayerNorm/RMSNorm/WeightNorm, residual connections, Pre-LN vs Post-LN, DeepNorm, and HybridNorm.
Feed Forward Networks
The MLP inside every transformer block: activations, initialisation, gated variants (GLU, SwiGLU, GeGLU), and why the FFN is where most of a transformer's parameters live.
Seminars
Student-led seminars, each on one topic that is also in the syllabus. Two decks are online with full slide-by-slide walkthroughs; the other four will land here as they are presented.
Linear RNNs
Recurrent models with linear state updates — the family that ties together linear attention, state-space models, and modern efficient sequence architectures.
Data Creation
A high-level overview of LLM-driven data creation: curation, synthesis (WRAP, Self-Instruct, Evol-Instruct), model collapse, decontamination, and evaluation.
Efficient Deployment
Serving LLMs cheaply — quantisation, batching, KV-cache tricks, and the deployment-time engineering that actually determines cost per token.
Test-Time Training
Networks that keep learning at inference — how a small optimisation loop per input can adapt weights to the specific example the model is looking at.
Diffusion Language Models
Language models that generate by iterative denoising instead of left-to-right sampling — parallel decoding, controllability, and the current tradeoffs versus autoregression.
Reasoning in LLMs
Chain-of-thought, self-consistency, verifiers, and the recent reasoning-model line — how (and how much) language models can actually reason.
Tutorials
Math for Deep Learning
A focused refresher on the linear algebra and calculus that show up in deep learning: vectors, matrices, and derivatives.
Talking Matrices
From word embeddings and SVD to Word2Vec, RNNs, transformers, and the BERT family.
Assignments
Transformers from Scratch
Build an encoder-decoder from nn.Linear, then compare positional encoding, attention,
normalisation, and tokenisation choices.
MoE, Optimizers, Decoding
Every line of the code with the shape of every tensor: the tensor toolkit, RoPE derived, Mixture of Experts, five optimisers from scratch, and greedy, top-k, top-p and beam search.
Project
Past-year questions
Mid Semester · 2023
Long-distance dependencies, ELMo, and encoder-decoder attention, with the official handwritten key for comparison.
Mid Semester · 2024
RNN dependencies, seq2seq attention, the transformer encoder block, and LoRA versus adapter fine-tuning.
§3How the pages read
Every lecture and tutorial begins with a quick map of the deck, then moves through the material slide by slide.
The slide. The original image, available at full size.
The notes. A clear explanation of the point being made and the context around it.
Extra help. Where it earns its place, you will find a worked formula, a common pitfall, or a link back to earlier material.
Related slides sometimes appear together when separating them would make the explanation harder to follow. Otherwise, it is one slide and the notes it needs.
Aha moments. The central insight to keep in mind.
A likely misunderstanding, called out before it becomes a habit.
The exact mechanism or math, set aside for a closer read when you need it.
A link to the page where the prerequisite idea is introduced in more detail.
§4Credits & attribution
Course material (the lecture slides and tutorial decks) is the work of Prof. Manish Shrivastava and TA Aparajitha Allamraju for the Advanced NLP course (M23CS7.501) at IIIT Hyderabad, Monsoon 2026. Slide images are reproduced here for study purposes only.
Past-year exam papers are sourced from the community VijayrajS/iiitprevpapers repository.
The explanatory notes are written by Arihant Tripathy as a study aid. Found an error? Please open an issue.