Walking the GPT-1 forward pass in full to pinpoint which pieces got re-engineered later is a great way to teach this, and the pre-norm over post-norm convergence is the cleanest example. Pre-norm won mostly because it keeps gradients stable enough to train deep stacks without the warmup gymnastics post-norm needed. Are you planning to cover why RMSNorm then displaced LayerNorm in most recent models, since that's the next step in the same story?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Walking the GPT-1 forward pass in full to pinpoint which pieces got re-engineered later is a great way to teach this, and the pre-norm over post-norm convergence is the cleanest example. Pre-norm won mostly because it keeps gradients stable enough to train deep stacks without the warmup gymnastics post-norm needed. Are you planning to cover why RMSNorm then displaced LayerNorm in most recent models, since that's the next step in the same story?