Leaderboard

We evaluate 9 state-of-the-art LLMs using a minimalist agent loop with stateless Bash execution and file editing tools. Each task allows up to 50 interaction rounds. Results are averaged over 4 independent runs (avg@4).

Abstract

While LLM-based coding agents have advanced to repository-scale engineering, existing benchmarks predominantly focus on software maintenance, overlooking the foundational capability of system construction.

We introduce CSBench, a benchmark evaluating project-level Spec-to-Code implementation across core computer science domains (e.g., OS, compilers, databases). CSBench features 100 expert-curated tasks from top-tier university assignments within containerized environments.

Evaluating 12 state-of-the-art LLMs reveals a stark performance disparity: even top-tier models like GPT-5.2 see success rates plummet from over 90% in algorithmic tasks to below 35% in system-level domains. Our analysis identifies a fundamental deficiency in systemic reasoning and construction knowledge as the primary bottleneck, where agents either terminate prematurely due to overconfidence or fail to translate execution feedback into valid architectural fixes.

Key Highlights

Benchmark Design

Each CSBench task is formally defined as a tuple T = (S, C_init, U, E): a Specification, Initial Scaffold, Test Suite, and a containerized Environment. Given (S, C_init), a model produces a completed implementation; a task is solved when all tests pass inside the Docker container.

Three-Stage Construction Pipeline

CSBench employs an adversarial, expert-in-the-loop workflow to transform heterogeneous courseware into standardized, execution-based benchmark tasks.

1

Sourcing

Candidate projects sourced from open-source CS courses at top-tier universities, organized by a comprehensive topic taxonomy (System, AI, Application, Others). Priority given to projects with non-trivial, multi-file starter scaffolds.

2

Annotation

Domain experts construct task artifacts with adversarial review: Annotators build scaffold code, specifications, and test suites; independent Reviewers audit outputs and iterate until strict criteria are met.

3

Quality Assurance

Three-layer verification: Automated checks (solvability, layout), AI-assisted auditing (leakage, ambiguity), and Human-in-the-loop decision (spec fidelity, runtime stability).

The three-stage construction pipeline of CSBench: Sourcing, Annotation, and Quality Assurance.

Domain & Language Distribution

CSBench spans 4 high-level topics and 19 sub-topics, ensuring systematic domain representation. It covers Python (41%), C/C++ (32%), and other mainstream languages including Java, JavaScript, Go, Rust, and Assembly.

Left: Task distribution by course topic taxonomy. Right: Programming language distribution across CSBench tasks.

Benchmark Case: CS000232

CS000232 is a concrete CSBench task instance included in this repository. The panel below exposes its complete folder structure so visitors can inspect the benchmark packaging, starter code, metadata, and test files directly on the website.

Detailed Results

Pass@k Analysis

Pass@k metrics show GPT-5.2 maintains a steady lead in AI and Application categories, while Gemini-3-Pro demonstrates superior performance in the System category, consistently outperforming GPT-5.2 across all k values.

Pass@k performance of representative models across different task categories.

Failure Analysis

We classify failed agent trajectories into four categories to diagnose root causes of failure in system construction tasks.

Failure Distribution

Strategic & Planning and Debugging & Iteration constitute the vast majority (>60%) of failures across all models, indicating that cutting-edge models generally handle tool invocation and code correctness well, but struggle with long-term reasoning and planning.

Distribution of failure categories across all task instances for each model.

Citation

If you find CSBench useful in your research, please cite:

@inproceedings{csbench2026,
  title   = {CSBench: A Comprehensive Benchmark for Evaluating
             Project-Level System Construction in Computer Science},
  author  = {Anonymous},
  booktitle = {Proceedings of the 43rd International Conference
               on Machine Learning (ICML)},
  year    = {2026}
}