Google DeepMind and Artificial Analysis have introduced MKA, the Multimodal Knowledge Agent benchmark, which is designed to test how well AI agents handle professional assignments whose decisive evidence sits inside blueprints, schematics, scans, charts, and similar visual files. The report argues that once layout is stripped away, many existing agent benchmarks collapse into text-only problems that models can solve from extracted tokens alone. MKA instead scores visual reasoning in combination with deliverable quality, execution cost, and tool use.
Inside the 200-task evaluation
In each MKA task, the agent is given a workspace of source files and asked to produce a finished deliverable, such as a report, spreadsheet, slide deck, or marked-up drawing. The assignments reproduce multi-step projects that seasoned professionals carry out as part of their regular work.
- Tasks: 200 scored assignments
- Coverage: 10 industry verticals and 83 occupations
- Expert effort: three to eight hours per assignment
- Typical outputs: reports, spreadsheets, presentations, and annotated drawings
- Scoring emphasis: 75% factual accuracy and 25% visual design
Specialists in each field authored the assignments, pairing realistic prompts with multimodal source files and a rubric containing roughly 70 weighted yes-or-no checks. Scoring therefore judges the submitted artifact on its own merits rather than matching it to one fixed reference answer.
Hard checks drive the score
Rubric criteria are split into two groups: Climb and Regression. The harder requirements fall under Climb and produce the headline metric, the Climb weighted satisfaction rate (Climb WSR). Regression criteria gauge baseline competence, an area where top models generally cluster together.
Scoring is handled by an automated grading agent that loads each submitted file, renders its pages, examines the formulas inside spreadsheets, and checks the result against the expert-written rubric. Artificial Analysis uses GPT-6 Luna, a non-Google model, at its medium setting as the judge. In the reported validation, its grading decisions matched those of domain experts 96% of the time.
One sandbox, 250 turns
Every task is executed through the public Stirrup harness inside a Linux sandbox capped at 250 turns. Within that environment, agents can run code, reach the internet, search the web, and fetch pages; models that accept image input also receive an image viewer. Source images keep their native dimensions unless a provider enforces its own limits.
To reduce benchmark leakage, Google keeps the 200 scored tasks private. A public sample of 10 tasks illustrates the file structure, prompts, and expected deliverable formats without exposing the evaluation set.
Small score gaps, large bills
Only four percentage points separate the three leaders in the reported Climb WSR ranking, yet their average execution costs diverge far more sharply:
- Claude Fable 5.2 (Max): 57% Climb WSR, about $32 per task on average
- Gemini 4 Carbon (High): 54% Climb WSR, about $6 per task on average
- Claude Opus 5.5 (Max): 53% Climb WSR, cost not specified
Several smaller or more economical configurations posted the lowest per-task costs:
- GPT-6 Luna (Max): $0.18 per task
- Claude Haiku 5.5 (Max): $0.32 per task
- MiMo-V2.6-Pro: $0.32 per task
- GPT-6.1 Sol: about $1 per task
One architectural-audit task shows how tool usage drives the cost spread. Claude Fable 5.2 reached 87% after 224 turns, 203 image views, roughly three hours of runtime, and about $68 in execution costs. Gemini 4 Carbon reached 72% after 76 turns, about one hour, and approximately $8.
Specialists split the field
Performance shifts considerably from one industry to the next, so each vertical has a different leader:
- Claude Fable 5.2: Software & Data, Engineering, Logistics & Mapping, Business, and Finance
- Claude Opus 5.5: Education and Construction
- Gemini 4 Argon: Science
- Gemini 4 Carbon: Healthcare
- Claude Sonnet 5.5: Law & Government
Rankings also move with output format. Scores are weakest for Word documents and plain-text files, with no evaluated model exceeding 52% on either, while PDFs, spreadsheets, and presentations produce higher results. Choosing a model for an agent product therefore hinges on the target industry and the artifact it must produce, not only on aggregate leaderboard position.
Where MKA helps
Since the evidence needed to finish every assignment is embedded in visual material, the benchmark directly probes whether an agent can interpret that material and carry it into a usable professional deliverable. The accompanying technical report also probes how performance changes when models are denied visual access, which helps isolate the contribution of multimodal perception.
Turn counts highlight a further deployment constraint. Evaluated models average between 30 and 130 turns per task, producing wide differences in latency and total inference cost. For agents in production, the report notes, those differences can matter more than modest gaps in per-token API pricing.
What the scores leave open
Several design decisions shape how far the results extend to production systems:
- The private 200-task set cannot be inspected independently, though keeping it hidden does reduce contamination risk.
- The autorater agrees with experts on 96% of grading decisions, so a small share of scores remains uncertain.
- The 250-turn cap, the sandbox, and the provided tools describe one specific operating environment that may not resemble a deployed agent stack.
- Cost numbers reflect model settings, tool usage, provider pricing, and runtime behavior, any of which can change.
Artificial Analysis publishes the full leaderboard alongside example outputs, and additional models can be evaluated under the same conditions through the public Stirrup repository.