Sahaj.blogs
23 Jun 2025 · Paper summary

Paper summary: Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights

Drag-and-Drop LLMs train a generator that maps a few unlabelled task prompts straight to LoRA weights, specialising an LLM for a new task in seconds without any gradient-based fine-tuning on it.

Figure from the paper "Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights"
Figure from Liang et al. (2025), licensed CC BY 4.0. (a) Parameter-Efficient-Fine-Tuning

Problem

LoRA (my February study notes) makes fine-tuning cheaper, but every new dataset still needs its own optimisation run.

Method

Drag-and-Drop LLMs (DnD) learn a prompt-conditioned parameter generator:

  1. a lightweight text encoder turns a batch of unlabelled task prompts into condition embeddings
  2. a cascaded hyper-convolutional decoder turns those into the full set of LoRA matrices

The generator is trained on many (prompts, trained-LoRA-checkpoint) pairs. For a new task, it produces task-specific weights in seconds, with no training on that task.

Key results

From the abstract:

  • up to 12,000× lower overhead than full fine-tuning
  • average gains up to 30% over the strongest training LoRAs on unseen common-sense, maths, coding and multimodal benchmarks
  • robust cross-domain generalisation without seeing target data or labels

Limitations

There’s no dedicated limitations section in the extracted text. The generator must first be trained on a diverse collection of existing LoRA checkpoints.

#llm#fine-tuning#peft

Sources

  1. Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (arXiv:2506.16406) — arXiv
  2. Hugging Face Daily Papers, 2025-06-23 — Hugging Face
  3. Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (arXiv HTML full text) — arXiv

This is a summary written from the linked public sources. Figures and numbers are as reported there.