Attention, Transformers, and LLMs: a hands-on introduction in PyTorch
In this 3-hour workshop, we will build a large language model from the ground up using PyTorch. This workshop focuses on the fundamentals of attention and the transformer architecture. We will assume a foundation in basic calculus, linear algebra, optimization and Python programming. Experience using PyTorch will be helpful, but is not strictly required.
Tentative agenda:
- Overview of the language modeling task
- Attention: Query, Key, and Value
- Self Attention
- Positional Encodings
- The Transformer Architecture
- Autoregressive Models
- Fitting our LLM
- Hugging Face: a high-level API for LLMs
- Conclusions
Live Workshop
No live sessions are currently planned for this workshop.
Resources
For a self-guided version, you can read the Attention, Transformers, and LLMs: a hands-on introduction in PyTorch Workshop notebooks on our workshop site.