Tokenization, pretraining, fine-tuning, RLHF, context windows, sampling temperature, hallucination, evaluation and alignment.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- What is tokenization in the context of language models?
Show answer
Breaking input text into smaller units (tokens) that the model can process- What is a token in NLP?
Show answer
A unit of text, typically a word or subword, that a language model processes- How does subword tokenization improve handling of rare words?
Show answer
It breaks rare words into smaller units that appear in the training vocabulary, avoiding out-of-vocabulary problems- ____ is a tokenization method that splits words into frequently used subword units
Show answer
Byte Pair Encoding- Why do language models require tokenization as a preprocessing step?
Show answer
It converts raw text into discrete units that can be mapped to embeddings and processed by the model- What is pretraining in the context of language models?
Show answer
Training a model on large unlabeled text corpora to learn general language patterns and knowledge- What is the primary objective of masked language modeling during pretraining?
Show answer
Predicting randomly masked tokens in a sequence using the surrounding context- What pretraining task where a model predicts the next token from previous tokens is called ____?
Show answer
causal language modeling- Why is pretraining on large datasets essential for modern LLMs?
Show answer
It allows the model to learn broad linguistic knowledge and general world facts before task-specific fine-tuning- At what scale are modern language models typically pretrained?
Show answer
On hundreds of billions to trillions of tokens- What is fine-tuning in machine learning?
Show answer
Adapting a pretrained model to a specific downstream task using labeled data- How does the learning rate for fine-tuning compare to pretraining?
Show answer
Fine-tuning uses a much lower learning rate to preserve previously learned knowledge- What is the purpose of instruction fine-tuning?
Show answer
Training the model to follow user instructions and produce more helpful, task-oriented responses- Name a parameter-efficient fine-tuning method that requires fewer trainable parameters than full fine-tuning
Show answer
LoRA (Low-Rank Adaptation)- What is catastrophic forgetting in the context of fine-tuning?
Show answer
Loss of previously learned knowledge when a model is aggressively fine-tuned on a small dataset
FlashKeepers