AI Agent UpdateBest AI agent scores just 12% on SemiAnalysis's new GDPx expert benchmark
SemiAnalysis's new GDPx benchmark gave the best AI agent a score of just 12% across 206 expert tasks — and every model scored zero on finance and legal work. The age of the all-capable agent is not here yet.
The News
SemiAnalysis released GDPx on November 2, a benchmark for general agentic capabilities spanning 11 professional domains — chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement, hardware design, materials science, alignment and software engineering — built with 14 specialized data partners across 206 tasks. On the version 1.0 leaderboard of 13 models, Anthropic's Claude Opus 5.5 leads at 12.0%, Google's Gemini 4 Carbon is second at 10.9%, and OpenAI's GPT-6 Astra is third at 9.6%. Meta's Muse Wonder Full ranks fourth at 9.0%.
The headline is the low ceiling, not the ranking. Even the best model fails roughly seven out of eight realistic expert tasks, and every model scored zero on finance and legal under pass-or-fail grading — meaning not one completed those tasks to the required standard. SemiAnalysis itself cautions that the headline scores are pulled down by safety classifiers in cyber, biology and alignment: filters that block outputs deemed risky also block legitimate expert work. Cost adds a second dimension to the ranking: Gemini 4 Carbon costs $16.29 per task, less than a third of Opus 5.5's $51.41, so the second-place model is far cheaper per attempt.
The benchmark lands at a moment when agent hype is at its peak — coding agents, autonomous workers, agents that act without being asked. GDPx is a cold-water test for that hype: it suggests today's frontier models are impressive assistants and unreliable experts. The scores are SemiAnalysis's own and have not been independently re-run, so treat the ranking as one respected lab's measurement, not a settled verdict. But as a directional signal, it is sobering for anyone pricing in near-term agent autonomy.
The best AI agent in the world just scored 12% on the expert test.
Why It Matters
This matters because benchmarks set expectations, and expectations set valuations. The AI industry's trillion-dollar question is when agents stop assisting experts and start replacing chunks of expert work. A respected benchmark answering 'not yet, by a wide margin' is a direct challenge to revenue forecasts that assume rapid agent deployment in finance, law and other high-value domains. If agents cannot pass those tests, the enterprise agent market grows more slowly than the pitch decks claim.
It matters for the model labs' competitive dynamics too. Claude leads on capability, Gemini leads on cost-efficiency, and the gap between first and third is just 2.4 percentage points — a tight field. That tightness means the next model generation, not the current one, will decide the ranking, and it means buyers can choose on price without giving up much capability. A market where the second-best model costs a third as much is a market heading for a price war.
The third reason is methodological honesty. SemiAnalysis published its caveats — safety classifiers dragging scores down, pass-or-fail grading producing zeros — rather than hiding them. That transparency is itself a sign of a maturing field: benchmarks that admit their limits are more useful than benchmarks that flatter their sponsors. Readers should still note the limit that matters most: one lab's scores, not yet independently reproduced.
Analyst's Take
The verdict is Mixed. The bearish read is obvious: 12% is a failing grade by any standard, and zero on finance and legal punctures claims that agents can already do expert work. The bullish read is subtler: a hard, honest benchmark is what a maturing field needs, and the cost gap — Gemini at a third of Claude's price per task — shows competition is already driving efficiency. Both reads are true at once: the technology is further from autonomy than the marketing suggests, and closer to useful deployment than the headline score implies.
What to watch next is the second version of the leaderboard and whether independent labs reproduce these scores. If scores climb quickly, the benchmark becomes a progress tracker; if they stall near 12%, the agent story shifts from 'autonomous workers' to 'supervised assistants' — a smaller, but still large, market. Reader poll: agents at 12% — overhyped or just early?
By The Numbers
FAQ
What is GDPx?
A benchmark — a standardized test — for general agentic capabilities, released November 2 by SemiAnalysis. It spans 11 professional domains, from chip design and cybersecurity to finance, legal work and software engineering, built with 14 specialized data partners across 206 tasks.
Who scored what?
Anthropic's Claude Opus 5.5 leads at 12.0%, Google's Gemini 4 Carbon is second at 10.9%, OpenAI's GPT-6 Astra is third at 9.6%, and Meta's Muse Wonder Full is fourth at 9.0%. Costs differ sharply: Gemini 4 Carbon costs $16.29 per task versus $51.41 for Opus 5.5.
Why are the scores so low?
SemiAnalysis itself cautions that safety classifiers — automated filters that block risky outputs — pull down scores in cyber, biology and alignment work, and that every model scored zero on finance and legal under pass-or-fail grading. The test is deliberately hard: realistic expert tasks, not quiz questions.
Have these scores been independently re-run?
No. The scores are SemiAnalysis's own and have not been independently re-run, a caveat stated here rather than buried.
Why do the tickers matter?
Anthropic and OpenAI are private, so no direct ticker exists. Google, Meta and Microsoft are named as read-throughs: Google builds Gemini, Meta builds Muse, and Microsoft is OpenAI's key partner and investor.
👁
Source: Unite.AI