eqho
/
LLM Lab
model
lab-tiny
the run
04 / 09
Lesson 04
Attention
Every token asks every earlier token: are you relevant to me?
Next: The transformer block
01
How big is a model?
02
Text becomes tokens
03
Tokens become vectors
04
Attention
Query, key, value
Score every pair
No peeking forward
Many heads at once
05
The transformer block
06
Predicting the next token
07
Training
08
From autocomplete to assistant
09
Inference in production
01 / 01
Query, key, value
Each token makes three versions of itself: what I'm looking for, what I contain, what I'll hand over.
←
→
space