RNNs, vanishing gradients, LSTM, GRU, seq2seq, attention precursors and embeddings.
35 cards · basic cards · AI-written, checked twice. Edit anything.
- What is a recurrent neural network (RNN)?
Show answer
A neural network with connections from hidden units back to themselves, allowing it to process sequential data by maintaining hidden state across time steps.- What is the hidden state in an RNN?
Show answer
The representation of information from previous time steps, passed forward to the next time step to capture sequential context.- What is parameter sharing in RNNs?
Show answer
The same weight matrices are used at every time step, allowing the network to generalize across sequence positions.- What types of data do RNNs process well?
Show answer
Sequential data such as text, time series, speech, and video frames where order and temporal dependencies matter.- What is an unrolled RNN?
Show answer
The representation of an RNN shown explicitly across multiple time steps, making the recurrent connections and parameter sharing visible.- What is a bidirectional RNN (BiRNN)?
Show answer
An RNN that processes sequences in both forward and backward directions, capturing context from both preceding and following positions.- What is the vanishing gradient problem?
Show answer
Gradients shrink exponentially during backpropagation through many time steps, making it hard to learn long-term dependencies.- Why does the vanishing gradient problem occur in RNNs?
Show answer
Backpropagation through time involves repeatedly multiplying gradients by weight matrices with values less than one, causing exponential decay.- What is the exploding gradient problem?
Show answer
Gradients grow exponentially during backpropagation through time, causing numerical instability and divergence during training.- What is gradient clipping?
Show answer
A technique that scales or caps gradients to prevent them from exceeding a maximum magnitude, addressing exploding gradients.- What signals exploding gradients during training?
Show answer
Loss becomes NaN or infinite, or weights become extremely large after a single update.- What is LSTM?
Show answer
Long Short-Term Memory, an RNN variant with gated mechanisms that enables learning of long-term dependencies while mitigating vanishing gradients.- What is the cell state in an LSTM?
Show answer
A separate memory pathway that runs through the LSTM, carrying long-term information with minimal transformation across time steps.- What does the LSTM forget gate do?
Show answer
It learns to selectively discard information from the cell state, deciding which past information is no longer relevant.- What does the LSTM input gate do?
Show answer
It controls which new information from the candidate value gets added to the cell state at each time step.
FlashKeepers