My RNN works fine on short sequences.
When I give it longer inputs, predictions become random.
Loss increases with sequence length.
It feels like the model forgets earlier information.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
My RNN works fine on short sequences.
When I give it longer inputs, predictions become random.
Loss increases with sequence length.
It feels like the model forgets earlier information.
This happens because standard RNNs suffer from vanishing and exploding gradients on long sequences.
As the sequence grows, important signals either fade out or blow up, making learning unstable. That is why LSTM and GRU were created.
Switch to LSTM or GRU layers and use gradient clipping:
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
Common mistakes:
Using vanilla RNNs for long text
Not clipping gradients
Too long sequences without truncation
The practical takeaway is that plain RNNs are not designed for long-term memory.