Paper summary: A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code

Problem

LLMs write more and more production code, but existing security benchmarks often test isolated snippets that don’t reflect real AI-assisted programming.

Method

A.S.E (AI Code Generation Security Evaluation) is a repository-level benchmark. Tasks mirror real programming work inside actual codebases, giving a more realistic test of whether generated code introduces security problems.

Key results

From the abstract, evaluating leading LLMs:

• current LLMs still struggle with secure coding
• repository-level complexity is hard even for models that do well on snippet-level tasks
• a larger reasoning budget does not necessarily lead to better code

Limitations

There’s no dedicated limitations section in the extracted text. Results reflect the models available when the benchmark was built.