AI updates
Gemma 2: 9B and 27B with sliding-window attention and distillation
Google released Gemma 2 in 9B and 27B sizes, trained on about twice the data of Gemma 1 and using sliding-window attention, logit soft-capping, knowledge distillation and model merging.
- Gemma 2 alternates sliding-window and full attention layers, and soft-caps logits for stability.
- The 27B model was trained on 13 trillion tokens, according to the post.