Paper Readings
Paper Readings
Reading closely rather than widely.
Transformer architecture
Attention Residuals: A Comprehensive Understanding
This paper addresses a fundamental problem in training deep transformer models: uncontrolled hidden-state magnitude growth as model depth increases. The authors propose Attention Residuals (AttnRes), which replaces standard...
More to come.