Paper summary: The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain

Problem

The authors argue that brain-like, scale-free networks generalise over time in ways current ML struggles with, and they want an LLM architecture that connects Transformers with models of the brain.

Method

BDH is built from n locally interacting neuron particles in a scale-free, biologically inspired network. It can be written as an attention-based state-space sequence model, with a GPU-friendly formulation. Its working memory during inference relies on synaptic plasticity with Hebbian learning (using spiking neurons): connections strengthen when the model processes a concept.

Key results

From the abstract:

• Transformer-like scaling: rivals GPT-2 on language and translation at the same parameter count (10M to 1B) and the same data
• specific synapses strengthen whenever BDH processes a specific concept
• sparse, positive activations, with monosemanticity demonstrated on language tasks
• the neuron interaction graph is highly modular with a heavy-tailed degree distribution

Limitations

There’s no dedicated limitations section in the extracted text. The comparisons are against GPT-2-scale models, not current large LLMs.