Paper summary: LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Problem

Extended context windows had topped out around 128k tokens. Fine-tuning at long lengths is costly, long training texts are scarce, and new, unseen token positions introduce “catastrophic” values in positional encodings.

Method

Three ideas around RoPE (rotary position embeddings):

1. Non-uniform interpolation: an efficient search finds how much to rescale each RoPE dimension, and at which positions. This gives a better starting point for fine-tuning, and alone enables about 8× extension without fine-tuning.
2. Progressive extension: first fine-tune a 256k-length model, then apply a second interpolation to reach 2048k.
3. Readjust at 8k so performance on the original short context window is recovered.

Key results

The abstract reports, for the first time, a 2048k-token context window for pre-trained LLMs, using up to only 1k fine-tuning steps at training lengths within 256k, while keeping performance on the original short window. Experiments cover LLaMA2 and Mistral.

Limitations

There’s no dedicated limitations section in the extracted text. Supporting a window doesn’t mean equal quality across it, and actually running a 2M-token context still has large compute and memory costs at inference.