Paper summary: Writing in the Margins: Better Inference Pattern for Long Context Retrieval

Problem

Long-context models can accept huge inputs, but they don’t always use the relevant parts well, especially for multi-hop reasoning or aggregating information spread through a document.

Method

Writing in the Margins (WiM) is an inference pattern, with no training:

1. Long inputs are already processed in chunks during chunked prefill of the key–value cache.
2. After each chunk, the model writes a short note, a margin, about what that chunk contains that’s relevant to the query.
3. Margins are classified as relevant or not, and the useful ones are added before the final answer is generated.

The computational overhead is small, and the margins can also give users ongoing updates during long retrieval.

Key results

From the abstract, on off-the-shelf models without fine-tuning:

• +7.5% average accuracy on reasoning tasks (HotpotQA, MultiHop-RAG)
• more than 30% F1 increase on aggregation tasks (CWE)

Limitations

There’s no dedicated limitations section in the extracted text. Generating margins adds some extra decoding per chunk, which the authors describe as a marginal overhead.