Paper summary: QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs

QeRL combines NVFP4 quantisation with LoRA for RL training of LLMs; beyond saving memory and speeding up rollouts, quantisation noise raises policy entropy and helps exploration. It enables RL on a 32B model with one H100.

Figure from the paper "QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs"

Figure from Huang et al. (2025), licensed CC BY 4.0. Figure 1: Rollout speedup and accuracy of QeRL on Qwen2.5-7B-Instruct. QeRL achieves faster RL rollout and end-to-end training speeds (batch=8), while delivering performance superior to vanilla LoRA and QLoRA, also …

Problem

RL is key to LLM reasoning, but it’s expensive: lots of GPU memory and long rollout phases.

Method

QeRL pairs NVFP4 4-bit quantisation with LoRA (my QLoRA study post covers the basic idea of 4-bit base weights plus adapters). The authors’ further finding: quantisation noise increases policy entropy, which improves exploration during RL. Adaptive Quantization Noise (AQN) tunes that noise during training.

Key results

From the abstract and Figure 1:

over 1.5× speed-up in the rollout phase
the first framework to enable RL training of a 32B LLM on a single H100 80GB
faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA
for the 7B model, matches full-parameter fine-tuning on GSM8K (90.8%) and MATH 500 (77.4%)

Limitations

From the paper: experiments cover 3B to 32B, so whether it holds above 70B is open. Because RL runs take tens to hundreds of hours, evaluation was limited to reasoning benchmarks.

Paper summary: QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs

Problem

Method

Key results

Limitations

Sources