Paper summary: The Unreasonable Ineffectiveness of the Deeper Layers

Problem

How is knowledge stored across an LLM’s layers? Pruning can test this: if removing a layer doesn’t change performance on knowledge-heavy QA benchmarks, that layer isn’t needed to store the knowledge those questions use.

Method

1. Measure similarity between the representations at different depths, and find the contiguous block of layers whose removal changes the representation least.
2. Remove that block.
3. “Heal” the model with a small amount of parameter-efficient fine-tuning (QLoRA). Each experiment fits on a single 40GB A100.

Key results

For some common open-weight models, performance on QA benchmarks shows minimal degradation until a large fraction, up to half, of the layers is removed. The authors give two possible readings: current pre-training doesn’t make good use of deeper layers, or shallow layers play a critical role in storing knowledge.

Limitations

There’s no dedicated limitations section in the extracted text. The finding is measured on question-answering benchmarks. Other abilities, such as reasoning or long-form generation, can depend on different layers.