Paper summary
Paper summary: Writing in the Margins: Better Inference Pattern for Long Context Retrieval
Writing in the Margins processes a long context chunk by chunk during prefill, writes short "margin" notes per chunk, keeps the relevant ones, and uses them to answer, improving off-the-shelf models without fine-tuning.
- Chunked prefill of the KV cache gives a natural point to ask the model for notes on each chunk at little extra cost.
- The margins can also be shown to users as progress updates during long retrieval.