Paper summary: Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

Problem

Coding agents must fold tool returns into ongoing reasoning. Standard left-to-right pre-training on code only shows this structure in one direction.

Method

The key observation: an agent’s action → observation → continuation loop is structurally like a function call site. A caller binds arguments, a callee returns a value computed elsewhere, and later code uses it. Ordinary code has this structure at internet scale.

Function-aware FIM mid-training masks whole functions, chosen via program-dependency-graph analysis and a “complexity-inferability” criterion, and trains the model to fill them in. The authors mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus from 968 GitHub repos, then apply existing agentic post-training pipelines.

Key results

From the abstract:

• SWE-Bench-Verified: +2.8 / +3.0 (7B/14B) and +3.2 (Qwen3-8B)
• SWE-Bench-Lite: +3.7 / +4.0 / +5.4
• holds across two post-training pipelines (R2E-Gym, SWE-Smith)
• reduces the capability erosion that agentic post-training otherwise causes on non-agent coding (LiveCodeBench) and tool-use benchmarks (tau-bench, BFCL)

Limitations

From the paper:

• the corpus and in-domain evaluation are Python-only
• the default recipe uses Gemini-3-Flash as a teacher for chain-of-thought, although self-generated rationales recover most of the gain
• cross-base validation comes from only one non-Qwen2.5 configuration