#data
- 3 Apr 2026 · Paper summary
Paper summary: DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
DataFlex unifies dynamic data selection, domain-mixture adjustment and sample reweighting in one framework built on LLaMA-Factory, so data-centric methods can be compared fairly and dropped into normal training.
- 11 Feb 2026 · Paper summary
Paper summary: OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
OPUS picks pre-training data at every iteration by projecting each candidate's optimizer-shaped update onto a target direction, at 4.7% extra compute; GPT-2 runs on 30B selected tokens beat full 200B-token training.
- 27 Jan 2026 · Paper summary
Paper summary: Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs
A survey of using LLMs to prepare application-ready data: cleaning, integration and enrichment, moving from rule-based pipelines to prompt-driven and agentic workflows.
- 23 Dec 2025 · Paper summary
Paper summary: DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI
DataFlow is an LLM-driven framework for building data-preparation pipelines from about 200 reusable operators with a PyTorch-style API, plus an agent that turns plain-language specs into pipelines.
- 6 Feb 2025 · Paper summary
Paper summary: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
SmolLM2 is a 1.7B model "overtrained" on about 11 trillion tokens in stages that mix web, maths, code and instruction data, with new datasets (FineMath, Stack-Edu, SmolTalk) built where existing ones were too small or noisy.
- 3 Jan 2025 · Paper summary
Paper summary: 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining
The authors turn 22,000 class hours of instructional videos into an interleaved image–text "textbook" corpus for vision-language pre-training, with denser knowledge than web-crawled interleaved data.
- 26 Sept 2024 · Paper summary
Paper summary: Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo is a family of open vision-language models trained on PixMo, new datasets collected without using any external VLMs, including detailed captions, free-form Q&A and 2D pointing data.
- 26 Jun 2024 · Paper summary
Paper summary: The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
FineWeb is a 15-trillion-token open pre-training dataset built from 96 Common Crawl snapshots, with every curation choice ablated; its 1.3T-token educational subset, FineWeb-Edu, gives large gains on MMLU and ARC.
- 21 Jun 2024 · Paper summary
Paper summary: Instruction Pre-Training: Language Models are Supervised Multitask Learners
Instruction Pre-Training augments raw pre-training text with synthesised instruction–response pairs (200M of them across 40+ task types) and reports stronger base models and better gains from later instruction tuning.
- 7 Jun 2024 · Paper summary
Paper summary: ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
ShareGPT4Video builds dense, precise video captions (40K with GPT-4V, then 4.8M with its own captioner) to improve both video understanding models and text-to-video generation.
- 26 Oct 2023 · Paper summary
Paper summary: A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
RECAP relabels a text-to-image training set with captions from a specialised captioning model instead of web alt-text, and reports better image quality and much better prompt following.
- 12 Sept 2023 · Paper summary
Paper summary: Textbooks Are All You Need II: phi-1.5 technical report
phi-1.5 is a 1.3B-parameter model trained mostly on synthetic "textbook-quality" data, reported to match models about five times larger on common-sense reasoning.
- 14 Aug 2023 · Paper summary
Paper summary: Self-Alignment with Instruction Backtranslation
Instruction backtranslation lets a language model label unlabelled web text with the instructions it answers, then pick the best pairs to fine-tune itself.