Paper summary
Paper summary: LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
LongLoRA extends the context window of pretrained LLMs cheaply by fine-tuning with shifted sparse local attention plus a LoRA variant that also trains embeddings and norms.
- Fine-tuning can use cheap local attention even though inference keeps full attention.
- LoRA works for context extension only when embeddings and normalisation layers are trainable too.