A benchmark of expert-written change requests
A November 2026 technical report from Toptal ExpertCrowd (arXiv:2611.13246v1 [cs.AI], 09.11.2026) introduces ExpertCrowd-SWE (EC-SWE), a benchmark of 100 expert-authored, multi-file software engineering tasks built to evaluate autonomous coding agents in large, unfamiliar codebases. The authors are Josip Vrdoljak, Akash Singh, John Lorenzo, Bas Steunebrink, Achille Nicoletti, Alex Yarotsky and Sam Pigott.
Each task is a change request written by a practising engineer against a pinned commit of a real repository. It is paired with a sealed reference solution and a task-specific grader whose weighted, mostly executable criteria award partial credit; must-pass blockers and pass/fail gates encode what a maintainer would refuse to merge. The tasks span 87 open-source projects and 11 languages. The median ticket is 148 words, while the median reference solution changes 471 lines across a median of 4.5 files. Every task image is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and an inert change score 0, and nearly half of the tasks ship alternative, lazy and adversarial control patches that the grader must classify correctly. The final 100 tasks were chosen for difficulty against current frontier models.
The scope is deliberately narrow: the benchmark asks whether an agent, working autonomously in a sandboxed, offline repository, can produce a change that an expert maintainer's grader accepts. The authors stress that the leaderboard compares models under shared task, prompt, sandbox and budget conditions; it is not a ranking of coding products or of overall model quality.
Why a new benchmark
The report argues that the dominant recipe, introduced by SWE-bench, mines merged fixes from public GitHub repositories and grades each task with the tests that shipped with the fix. According to the authors this has two weaknesses: the fixes, discussions and tests are public and plausibly present in pretraining corpora, so a high score can reflect recall rather than problem solving; and inherited tests were written to confirm one particular patch, so they can reject valid alternatives or accept incomplete ones.
The report recounts two 2026 events that made these issues hard to ignore. In February, OpenAI stopped reporting SWE-bench Verified after an audit found material issues in the tests or problem statements of 59.4% of 138 audited problems, together with evidence that frontier models had memorized gold patches. In July, a follow-up audit estimated that roughly 30% of SWE-Bench Pro tasks were broken, and OpenAI retracted its earlier recommendation of that benchmark, calling for evaluations built with experienced software developers and human oversight throughout.
Benchmarks with authored rather than mined tasks address contamination, but the authors note that the strongest agents now crowd the top of those leaderboards too: on DeepSWE v1.1, Claude Opus 5.5, GPT-6 Astra, DeepSeek V4.1 Flash and Gemini 3.8 Flash sit within 0.9 points of one another (72.3% to 71.4%). EC-SWE is positioned as a benchmark of changes no model has seen solved, written for the benchmark by engineers who know the codebase, never published as fixes, graded the way a maintainer reviews a pull request, and kept mostly private.
How the tasks are produced
Task authors are engineers from Toptal's talent network, described as more than 30,000 vetted and specialized experts, who apply to one of four areas: compilers and language tooling; systems, backend and networking; database and storage systems; and SRE, DevOps and infrastructure. Thirty-nine experts contributed to task authoring; after quality control, verification and selection, the final 100 tasks come from 17 of them.
Authoring follows a two-stage, draft-first workflow. In the draft stage the author submits the repository and the exact commit hash it is pinned to (never a branch name), the prompt written as a ticket, and a grading plan naming the checks and which must pass; a reviewer approves or returns it, and drafts are rejected for being unreadable, prescribing an implementation, lacking a grading plan or relying only on an AI judge, or proposing a problem the author has not solved. In the build stage the author implements the task on the HUD coding template, confirms that the reference solution scores 100% while an unchanged checkout scores 0%, and runs the task five times against a frontier model (Claude Opus 5, step limit 100); five zero runs are treated as a possible sign of a broken grader. AI assistance is permitted for environment wiring but not for the task idea or the reference solution. Recorded reasons for rejecting submitted tasks include reference solutions that were never tested (4), weak, broken or opinion-only graders (3), tasks or repositories that were too small (3), incomplete submissions (2) and one impossible task.
Tasks come from four routes under common contamination rules: never base a task on a bug already fixed in public, never use employer or client code, keep the task's own change unpublished, and use proprietary code only when the author owns it and grants the rights. The routes are changes based on expert experience on public repositories, tasks motivated by long-standing unresolved upstream issues, forks with expert-added problems, and codebases the expert owns. In the released set, 98 tasks are new changes on public open-source projects with no added substrate; one task, a PostgreSQL foreign-key migration preflight in SQLAlchemy, adds an expert-authored substrate patch of about 250 lines; and one, a payments webhook pipeline, is built on a codebase owned by its author. Base commits are recent: 82 of the 100 were committed in August or September 2026, with a median date of 04.09.2026, and at least 97 lie on the upstream default branch.
Each task ships four artifacts: a prompt consisting of the author's ticket plus an environment preamble (repository location, roughly two vCPUs and a hard memory limit, an offline sandbox, how to start backing services the repository's own tests need, and the instruction that hidden tests will grade the work and existing tests must not be modified); a Docker environment containing the repository at the pinned commit with its history vaulted, toolchains and a pre-populated dependency cache, with agent shells running as an unprivileged user and all answer-shaped files under a root-only directory; the author's reference solution (task/gold.diff), sealed inside the image; and a grader that executes hermetically, starting its own instance of any needed backing service such as Redis or a database on a private port.
Grading
Each grader is a list of named criteria with weights summing to exactly 1.0 across all 100 released tasks. Most criteria are executable checks of observable behaviour through the library's or service's own public interface; implementation-internal checks are avoided so that any reasonable solution can earn full credit. Three rule types combine: weighted criteria whose scores sum into partial credit; blockers, must-pass criteria that cap a run's score below full credit (for example at 0.60) rather than zeroing it; and gates, pass/fail preconditions with no weight that zero the run if violated, such as requiring that the repository's own test suite is green and untouched. Fourteen released tasks use gates.
The authors justify partial credit with their own data: only 1.0% to 3.8% of runs from Gemini 3.8 Flash, Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 earn full credit, so binary grading would compress all four into a 1% to 4% band, whereas partial credit spreads them across 32.2% to 40.0%. Ten tasks include an LLM-judged criterion carrying 2% to 25% of the task weight, 8.5% on average among the ten and 0.85% of total benchmark weight; the authors say LLM-judged criteria were kept under 10% of the total grade because of potential bias and reliability issues, and the remaining 90 tasks are graded entirely by deterministic checks.
Quality assurance and contamination controls
Quality is enforced at build time, through control patches, and through senior-engineer review. The image build fails unless the reference solution scores exactly 1.00 with every criterion passing individually and both the untouched checkout and an inert, comment-only change score 0.00 with every criterion failing. Explicit control patches must be classified correctly: alternative valid solutions that must pass (35 tasks), adversarial or cheat patches that must fail, for example ones that edit tests, special-case graded inputs or disable the feature under test (30 tasks), and lazy or no-op patches that must score low (42 tasks). Every task was quality-checked by a senior engineer, who examined its prompt, environment, reference solution, grader and diagnostic frontier-model runs; only tasks that passed received the platform status "verified".
On contamination, the authors report that each task's change is unpublished; 76 of the 100 tasks are held out; and every task was screened for overlap with 499,647 task records from 93 public software engineering benchmark datasets, by matching repository commits and task identifiers and by semantic similarity of prompts and code, with an LLM judge reviewing the closest pairs. They state that no task duplicated a public task or graded the same behaviour, although a minority touch the same component as a public task with a different change; upstream repositories are also monitored for changes overlapping reference solutions.
Evaluation setup
The authors evaluate seven models, five runs per task each, for 3,495 scored runs collected between 01.10.2026 and 10.10.2026. Every run uses the HUD hosted runner and the HUD SDK (hud-python, v0.6 series; latest release v0.6.21): the SDK's agent loop calls each model's API directly, exposes the tool schema that provider's models are trained on, and executes every tool call in the same task container; no vendor product such as Claude Code, Codex CLI or Gemini CLI is involved. All models share the task, prompt, container, resources, 1,000-step budget and grader, differing only in tool interface and agent loop, a deliberate contrast with DeepSWE's single bash tool for every model that the report itself flags as a limitation. Each model ran with extended reasoning at the platform default high-effort setting and no system prompt.
The evaluated configurations are: Claude Opus 5.5 and Claude Fable 5.1 via the Anthropic Messages API with bash and str_replace_based_edit_tool at effort max and a 128,000-token output limit; GPT-6 Astra via OpenAI Responses with a patch-based shell at effort max and 64,000 tokens; Gemini 3.8 Flash via Gemini generateContent with run_shell_command, read_file, replace, write_file, grep_search, glob and list_directory at thinking high and 65,536 tokens; and Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 via an OpenAI-compatible chat interface with bash, read, edit, write, grep and glob at effort xhigh, with 131,072 output tokens for the first two and the provider default for Grok 4.6.
Runs were excluded and rerun if invalidated on the platform, failed for infrastructure reasons, were smoke checks, or hit the step or output-token limit; limit-hit exclusions were concentrated in Claude Opus 5.5 (27 runs) and Gemini 3.8 Flash (55 runs), plus 2 Grok 4.6 runs. Gemini 3.8 Flash has no valid run on one released task and is scored over 99 tasks.
The report publishes three uncertainty measures: task-sampling intervals used for the headline scores, paired bootstrap and sign-flip permutation tests for model differences (2,000 resamples; 20,000 permutations, Holm-adjusted), and run-to-run intervals over the five run slots for comparability with DeepSWE and Terminal-Bench. Task-sampling intervals (plus or minus 3.3 to 5.5 points) are two to ten times wider than run-to-run intervals (plus or minus 0.4 to 1.9), which the authors take as evidence that leaderboards reporting only run-to-run noise understate how much rankings depend on the task sample.
Results
On the 100 released tasks, the report gives these mean task scores with 95% task-sampling intervals: Claude Opus 5.5, 67.3 ± 4.7; GPT-6 Astra, 63.7 ± 5.5; Claude Fable 5.1, 46.5 ± 3.7; Gemini 3.8 Flash, 40.0 ± 4.3 (99 tasks); Qwen 3.8 Max, 33.5 ± 3.3; DeepSeek V4.1 Flash, 32.5 ± 3.7; and Grok 4.6, 32.2 ± 3.5. The mean across all seven models is 45.1%. Counting only each model's best run out of five would lift the top three to 80.9%, 71.4% and 66.7% respectively.
Paired comparisons resolve four tiers. Against Claude Opus 5.5, the permutation test gives p = 0.20 for GPT-6 Astra and p < 0.001 for every other model. The top two are not separable: Opus leads by 3.6 points with a paired 95% interval from −1.4 to +9.7, winning 53 tasks to Astra's 41 with 6 ties. Both are clearly ahead of Claude Fable 5.1, which Opus outscores on 82 of 100 tasks. Fable leads Gemini 3.8 Flash by 6.5 points (+1.6 to +10.5), Gemini leads Qwen 3.8 Max by 6.5 (+3.0 to +11.0), and the lower three cannot be distinguished from one another.
Full-credit rates diverge sharply: 26.6% of Claude Opus 5.5 runs and 23.4% of GPT-6 Astra runs score 1.0, against 8.6% for Claude Fable 5.1 and at most 3.8% for the rest (1.0% for Qwen 3.8 Max and Grok 4.6). Opus earned full credit at least once on 47 tasks and on all five runs on 9; Astra did so on 36 and 15 tasks respectively.
The authors report substantial headroom: no task is fully solved by all seven models; on 80 tasks no model earns full credit on all five runs; both leaders score 100% on the same four tasks and both fall below 60% on 20. Task averages across models range from 2.5% to 69.5% (median 46.2%), and on 9 tasks even the best model averages below 50%. Of the 699 model-task pairs, 26 earn full credit on every run and 14 earn zero on every run. The leaders are complementary rather than ordered: their task scores correlate only moderately (Pearson r = 0.42), and on 14 tasks they differ by 40 points or more, 8 in Opus's favour and 6 in Astra's.
Some tasks defeat everyone. In an OAuth2 Proxy task on session and CSRF cookie accounting no model averages above 21.8%; in a Harbor quota-ownership task the best model averages 19.7%; and on the hardest task, a SystemVerilog export for the CirC compiler, all seven models land between 2.3% and 2.7%.
Reliability, lost credit and effort
The leaders differ in consistency: GPT-6 Astra's runs vary less within a task (mean within-task standard deviation of 7.3 points versus 13.6 for Claude Opus 5.5), and it reaches full credit on all five runs for 15 tasks against Opus's 9, while Opus reaches full credit at least once on more tasks (47 versus 36). Claude Fable 5.1 is the least consistent model (within-task SD 16.9) even though its run-to-run benchmark score is the most stable (±0.4). The authors suggest a user who can sample and verify several attempts would favour Opus's breadth, while one needing a single attempt to be right would value Astra's reliability.
Across all 3,495 scored runs, 274 (7.8%) were capped by a failed blocker and 80 (2.3%) were zeroed by a failed gate. These rules cost the strongest models the most: without blocker caps and gates, Claude Opus 5.5 would gain 3.7 points and GPT-6 Astra 2.9, while the lower group would gain 1.1 to 2.9; the ranking would not change. The report highlights an Asynq task on how task cancellation interacts with handler outcomes, where Claude Opus 5.5 and Grok 4.6 score zero on every run while DeepSeek V4.1 Flash averages 75.6%, as a case where gates catch behaviour that test-only grading would reward.
Effort varies widely and barely tracks score. Median agent steps per run range from 33 for GPT-6 Astra to 166 for Gemini 3.8 Flash, with five of seven models at a median of 127 or more (160 for Claude Opus 5.5); Astra reaches 95% of Opus's score with about a fifth of its steps. Within each model, the correlation between step count and score is weak (r between −0.15 and +0.17). Median container lifetimes range from 36 minutes for GPT-6 Astra to 100 minutes for Qwen 3.8 Max, the longest despite a mid-range step count.
Limitations and next steps
The report lists four limitations. First, each model used its provider's native tool interface through the HUD SDK, which is closer to how the models are trained and deployed than a shared bash tool but means part of any gap could reflect tool design rather than capability; a controlled comparison under a single harness is future work. Second, each model ran at its platform default high-effort reasoning setting, which was not swept. Third, 100 tasks yield intervals of plus or minus 3.3 to 5.5 points, so differences smaller than about 5 points between adjacent models should not be over-interpreted. Fourth, task changes are unpublished and screened, but the repositories themselves are public and likely in training data: EC-SWE tests problem solving in familiar-looking code, and released public tasks will eventually enter training corpora, which the 76 held-out tasks mitigate.
Comparing against DeepSWE v1.1, where four of the same models fall within 0.9 points of one another, the authors argue EC-SWE resolves differences existing leaderboards do not; those four span 34.8 points here, and DeepSeek V4.1 Flash ranks third on DeepSWE and sixth on EC-SWE. They caution that the comparison is indicative rather than controlled because the benchmarks differ in metric, harness and task population.
Planned future work includes auditing graders with an independent judge on a sample of agent runs and publishing disagreement rates; rerunning all models under a shared single-tool harness; sweeping reasoning effort; growing the corpus through further authoring waves and periodically retiring public tasks; and extending the same expert-authored, rubric-graded methodology to other professional domains, starting with management consulting.
Availability
The authors release 24 tasks with their environments, graders and results via the Hugging Face dataset at huggingface.co/datasets/ExpertCrowd/ec-swe, and hold out 76 to limit contamination. A public leaderboard is hosted at toptal.com/expertcrowd/benchmarks/expertcrowd-swe-benchmark.