Paper summary: CodeFusion: A Pre-trained Diffusion Model for Code Generation

Problem

The abstract’s analogy: imagine a developer who can only change the last line of code they wrote. Autoregressive code models are in that position. Once a token is generated, it’s fixed, so an early mistake can only be fixed by starting over.

Method

CodeFusion applies diffusion to code. Starting from noise, it iteratively denoises a complete program, conditioned on an encoding of the natural-language request. Every denoising step can revise any part of the program. The model is pre-trained and then evaluated on three languages: Bash, Python, and Excel conditional-formatting rules.

Key results

According to the abstract, CodeFusion with 75M parameters performs on par with state-of-the-art autoregressive systems of 350M to 175B parameters on top-1 accuracy, and outperforms them on top-3 and top-5. The authors attribute this to a better balance between the diversity and quality of its candidates.

Limitations

The paper doesn’t have a dedicated limitations section in the extracted text. The evaluation covers three targets (Bash, Python and Excel conditional-formatting rules), so results on other languages and longer programs aren’t covered by this paper.