Paper summary
Paper summary: LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
Tokens we treat as filler (punctuation, articles, stop words) carry a surprising share of an LLM's contextual memory; removing them hurts MMLU and long-context scores, and the paper releases a toolkit to measure this.
- Deleting "unimportant" tokens like commas and articles consistently degrades performance, even when the deleted tokens are irrelevant to the question.
- Contextualisation correlates strongly with how linear the layer-to-layer transformation is.