Paper summary: DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

Problem

Data selection, mixture optimisation and reweighting methods are developed in isolated codebases with inconsistent interfaces, which hinders reproducibility and fair comparison.

Method

DataFlex, from the same group as DataFlow (Dec 2025 on this blog), is built on LLaMA-Factory (Mar 2024). It supports three paradigms:

• sample selection
• domain mixture adjustment
• sample reweighting

It stays compatible with the normal training workflow as a drop-in replacement, and unifies shared operations (embeddings, inference, gradients), with support for DeepSpeed ZeRO-3.

Key results

From the abstract:

• dynamic data selection consistently beats static full-data training on MMLU for Mistral-7B and Llama-3.2-3B
• for mixtures, DoReMi and ODM improve MMLU accuracy and perplexity over default proportions when pre-training Qwen2.5-1.5B on SlimPajama at 6B and 30B tokens

Limitations

There’s no dedicated limitations section in the extracted text. It’s a framework, so its value grows with the number of methods it integrates.