Self-attention, multi-head attention, positional encoding, encoder and decoder stacks, BERT versus GPT and scaling laws.
About this set
40 basic cards for College Machine Learning. Set 4 of 4 in Deep Learning. It opens with "Self-attention mechanism" and "Q, K, V in self-attention". Study it alongside the other College sets below.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- Self-attention mechanism
Show answer
Each element attends to all elements and itself to compute representations- Q, K, V in self-attention
Show answer
Query, Key, and Value matrices computed from the input- Attention weight formula
Show answer
softmax(Q * K^T / sqrt(d_k)) * V- Why divide by sqrt(d_k) in attention
Show answer
Prevents dot products from growing too large and causing vanishing gradients- Self-attention output
Show answer
Weighted sum of value vectors, weighted by softmax attention scores- Multi-head attention
Show answer
Multiple parallel self-attention operations, each learning different representation aspects- Purpose of multiple attention heads
Show answer
Different heads attend to different positions and representation subspaces- Typical number of attention heads
Show answer
8, 12, or 16 heads- How attention heads are combined
Show answer
Concatenate outputs and project through a linear layer- Positional encoding
Show answer
Information added to embeddings to represent position of each element in sequence- Why positional encoding is needed
Show answer
Self-attention is permutation-invariant, cannot distinguish word order without it- Original Transformer positional encoding type
Show answer
Sinusoidal functions using sin and cos at different frequencies- Learned vs fixed positional encoding
Show answer
Original Transformer uses fixed sinusoidal values, not learned parameters- Two main Transformer components
Show answer
Encoder stack and decoder stack- Encoder stack function
Show answer
Processes input sequence through self-attention and feed-forward networks
FlashKeepers