Sahaj.blogs

Day page

1 Oct 2025

Study

Transformers · Attention Is All You Need

Reading the 2017 Transformer paper: dropping recurrence entirely, building everything from attention and feed-forward layers, and why that parallelises so well.

  • The Transformer removes recurrence: every position attends to every other position in one step.
  • Without recurrence, all positions in a sequence can be processed in parallel during training.

Read the post →

Paper summary

Paper summary: The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain

The Dragon Hatchling (BDH) is a brain-inspired language-model architecture of locally interacting "neuron particles" with Hebbian working memory, reported to match GPT-2 at the same size and to be more interpretable.

  • BDH's working memory at inference comes from synapses that strengthen with use (Hebbian learning), not from a KV cache.
  • The authors report GPT-2-like performance from 10M to 1B parameters on the same data.

Read the post →