The Problem with Current Coding Benchmarks

Most benchmarks for LLM coding agents — HumanEval, SWE-bench, Terminal-bench — share a fundamental blind spot: they only test whether an agent can produce a one-shot fix. Pass the test suite once, and you’re done.

But real software isn’t built in a single pass. It evolves over months and years through requirement changes, feature additions, and refactoring. Maintenance activities account for 60-80% of total software lifecycle costs. An agent that hard-codes a brittle fix and one that writes clean, extensible code may both pass the same test suite — but their long-term impact on the codebase is vastly different.

This is where SWE-CI (SoftWare Engineering — Continuous Integration) enters the picture. Published by researchers from Sun Yat-sen University, Alibaba Group, and Skylenage, SWE-CI is the first repository-level benchmark built on the Continuous Integration loop, shifting evaluation from static, short-term functional correctness toward dynamic, long-term maintainability.

The key insight is straightforward: maintainability can be revealed by tracking how functional correctness changes over time.

Paper: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration GitHub: SKYLENAGE-AI/SWE-CI Dataset: HuggingFace

Snapshot vs. Evolution: A Paradigm Shift

Existing benchmarks follow what the paper calls a snapshot-based evaluation paradigm: the agent gets a base codebase and a “golden” target, generates a single patch, and is scored on whether tests pass. The requirement is static — derived once from the gap between base and target.

SWE-CI introduces an evolution-based evaluation paradigm instead. The requirement is derived dynamically from the current state of the codebase (which the agent itself has been modifying), and the codebase updates iteratively. This means earlier design decisions compound — good architecture enables future changes, while accumulated technical debt makes each subsequent modification harder.

Snapshot-based vs. Evolution-based evaluation — SWE-CI proposes iterative evolution where earlier decisions compound into later outcomes

This distinction matters enormously. Under snapshot evaluation, an agent that produces spaghetti code with the right output is indistinguishable from one that writes well-structured, maintainable code. Under evolution-based evaluation, the spaghetti code agent will see progressively declining performance as its technical debt accumulates.

How SWE-CI Is Built

The benchmark comprises 100 tasks, each drawn from a real-world Python repository with a development history spanning an average of 233 days and 71 consecutive commits. The data curation pipeline involves four steps:

The 4-step SWE-CI data curation process: Repository Collection, Commit Span Extraction, Environment Construction, and Case Filtering

Step 1: Repository Collection

Unlike SWE-bench (which draws from a handful of well-known projects), SWE-CI casts a wide net across all Python repos on GitHub. Filtering criteria: 3+ years of active maintenance, 500+ stars, configuration/dependency files, unit tests, and a permissive license. This yields 4,923 repositories.

Step 2: Commit Span Extraction

For each repo, the main branch history is linearized. The pipeline identifies maximal subsequences where dependencies remain unchanged, forming candidate base/oracle pairs. Pairs with fewer than 1,000 modified lines are discarded. This yields 8,311 candidates.

Step 3: Environment Construction

Docker environments are automatically generated for each candidate pair. A self-repair mechanism detects missing dependencies and dynamically injects them. After this step, 1,458 viable pairs remain.

Step 4: Case Filtering

Three rounds of filtering ensure quality: running oracle tests against the base codebase, comparing test reports between base and oracle (requiring at least 5 test differences), and finally ranking by time span and commit count to select the top 100.

Each sample ships with complete source code and a pre-built Docker environment for reproducibility.

The Dual-Agent CI Loop

SWE-CI employs an Architect-Programmer dual-agent protocol that mirrors how professional software teams operate:

The Architect-Programmer dual-agent workflow in SWE-CI mimics real-world CI loops

The Architect Agent

Starting from failing tests, the Architect:

  1. Summarizes — reviews all failing tests and identifies patterns
  2. Locates — examines source code and attributes failures to specific deficiencies
  3. Designs — produces an incremental requirements document (max 5 items per iteration) using high-level natural language

The Programmer Agent

Given the requirements document, the Programmer:

  1. Comprehends — translates high-level requirements into explicit code specifications
  2. Plans — determines the programming effort needed
  3. Codes — implements the changes

This loop repeats for up to 20 iterations, with each cycle producing an updated codebase that feeds into the next iteration’s test comparison.

EvoScore: Measuring Maintainability

SWE-CI introduces EvoScore — a metric that captures long-term code maintainability as a single scalar. It builds on two concepts:

Normalized Change

Rather than a binary pass/fail, normalized change a(c) measures the fraction of the test gap the agent has closed (or widened). It falls in the range [-1, 1]:

  • a(c) = 1: agent has closed the entire test gap
  • a(c) = 0: no change from baseline
  • a(c) = -1: agent has broken every initially passing test

The Future-Weighted Mean

EvoScore aggregates normalized changes across all iterations using a future-weighted mean with parameter gamma. When gamma >= 1, later iterations receive greater weight — rewarding agents that maintain code quality over time rather than those that rush to pass early tests but accumulate technical debt.

This design directly mirrors the ISO/IEC 25010 definition of maintainability: code that remains easy to modify as evolution progresses.

Key Findings

The researchers evaluated 18 models from 8 providers, consuming over 10 billion tokens. Here are the headline results:

Finding 1: LLM Code Maintenance Is Improving Rapidly

Within every provider family, newer models consistently achieve higher EvoScores. Models released after 2026 show markedly larger gains than predecessors, suggesting the field is rapidly evolving beyond static bug-fixing toward sustained, long-term code maintenance.

EvoScore variation of state-of-the-art models from 8 providers on SWE-CI — newer models within each family consistently score higher

The Claude Opus series maintained a commanding lead throughout the observation period, with GLM-5 also emerging as a strong performer.

Finding 2: Providers Differ on Short-Term vs. Long-Term Focus

By varying the gamma parameter in EvoScore, the researchers reveal different strategic tendencies:

  • Long-term focus (higher gamma favors later iterations): MiniMax, DeepSeek, and GPT show preference for long-term gains
  • Short-term focus: Kimi and GLM lean toward immediate returns
  • Stable across settings: Qwen, Doubao, and Claude remain relatively consistent regardless of gamma

Model rankings shift as gamma changes — revealing which providers prioritize short-term vs. long-term code quality

Finding 3: Regressions Remain a Critical Weakness

This is the most striking result. The zero-regression rate measures what proportion of tasks an agent completes without ever breaking a previously passing test. Most models achieve a zero-regression rate below 0.25. Only two models in the Claude Opus series exceed 0.50.

Zero-regression rates across all evaluated models — most fall below 0.25, indicating frequent regressions during long-term maintenance

This means that even the best current LLMs, when tasked with sustained code evolution over dozens of iterations, will frequently introduce regressions — breaking functionality that previously worked. This is a critical gap between current AI coding capabilities and what’s needed for autonomous long-term software maintenance.

Why This Matters for Security and DevOps

From a security and software reliability perspective, SWE-CI’s findings carry important implications:

  1. Regression = vulnerability surface: Every regression an AI agent introduces is a potential security issue. If an agent inadvertently removes input validation or breaks an authentication check while implementing a new feature, the result is a vulnerability — not just a bug.

  2. CI/CD pipeline trust: Organizations increasingly considering AI agents in their CI/CD pipelines need to understand that current models still struggle with the maintenance aspect. Automated code generation is improving, but automated code stewardship is still nascent.

  3. Technical debt as attack surface: The finding that some models prioritize short-term fixes over long-term code quality means AI-generated technical debt could accumulate rapidly — creating increasingly fragile systems that are harder to secure.

  4. Testing is non-negotiable: SWE-CI’s architecture — where every iteration runs the full test suite — reinforces what security engineers already know: comprehensive testing is the only reliable way to catch regressions, whether introduced by humans or AI agents.

Conclusion

SWE-CI represents a meaningful step forward in how we evaluate AI coding agents. By shifting from one-shot snapshots to iterative evolution, it makes visible what was previously invisible: whether an agent writes code that lasts.

The current state of affairs is humbling. Even the best models — burning through billions of tokens — struggle to maintain code quality across extended evolution. Regressions remain pervasive. The gap between “code that works right now” and “code that continues to work as the system evolves” is still significant.

For anyone integrating AI agents into their development workflow, SWE-CI’s message is clear: writing code that works is only half the challenge. The other half is writing code that keeps working.

References