Paper summary: LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

Problem

Training on long sequences is expensive because self-attention cost grows quadratically. The paper’s example: a context of 8,192 needs 16× the attention compute of 2,048. So extending a pretrained model’s context window usually means heavy fine-tuning.

Method

Two ideas, combined:

1. Shifted sparse attention (S²-Attn): during fine-tuning, split the sequence into groups and attend only within each group. In half of the attention heads, shift the groups by half a group, so information still flows between neighbouring groups. The authors say this takes about two lines of code, and that the model still uses normal full attention at inference.
2. Improved LoRA: standard LoRA alone didn’t extend context well. It works once the embedding and normalisation layers are also trainable.

Key results

The paper reports strong results on Llama 2 from 7B to 70B. One example: extending Llama 2 7B from 4k to 100k tokens, or 70B to 32k, on a single machine with 8 A100 GPUs, while keeping the original architecture.

Limitations

The approach is about fine-tuning efficiency. The extended models still pay full long-context attention cost at inference. The conclusion reports a “minimal accuracy compromise” compared with full fine-tuning, not zero.