Real-World Web-Agent
Task Benchmark

Accuracy against cost

Scores are over both public datasets (384 calls per model), scored strictly. "Format tolerated" also counts a correct choice written after an explanation; only Claude Sonnet 5.5 moves noticeably.

Cost for OpenRouter models is the charge OpenRouter reported for each call. Cost for Sophtron is the on-demand price of an AWS g6e.xlarge (one NVIDIA L40S, us-west-2, $1.861 per hour) times the measured time of each call divided by estimated 15 concurrent jobs. See "How cost is counted" below.

Time per call

Time is measured from sending the request to receiving the complete answer. Hosted Qwen with thinking on is the outlier at 21 s.

Against each frontier model

Differences in correct answers are counted over the same 384 calls. Cost and time are ratios of the means above.

The task

Getting a statement from a website is a chain of small decisions: is this the right page, what to click next, which row or field holds the data, is the job done. Each recorded decision is a fixed question with a fixed list of options. Replaying the records lets us compare models on identical inputs without signing in to any site.

# illustrative example, not from the dataset
task: download the last 6 monthly statements
page: account overview (outline of headings, links, buttons)
question: which control should be used next?
[c1] link "Pay bill"
[c2] link "Statements & documents"   <- acceptable
[c3] link "Profile"
the model must answer with one option id, e.g. {"choice": "c2"}

A call counts as correct when the model returns an offered option that is in the accepted set. A reply that adds an explanation around the JSON counts as a format error under the strict score. Network failures count as wrong and are listed separately.

Results by dataset

Regression set: 70 cases from a code host, a utility and a retailer, three runs each

Two cases accept every option. They were run but not scored, so each model has 204 scored calls of 210.

New institutions: 60 cases from 39 sites, three runs each

Setup

• Same input for every model. The same system prompt, page state, question and options, and an output limit of 4,096 tokens including reasoning. Answer labels never enter a request.
• Frontier models (Claude Opus 5.5, Claude Sonnet 5.5, GPT-5.6 Terra) ran through OpenRouter with reasoning enabled at OpenRouter's default effort. No reply hit the output limit.
• Qwen3.8-27B ran through OpenRouter, pinned to DeepInfra at bf16 with thinking on, so every call used the same weights and precision.
• Jev 1.13 ran through OpenRouter's classification interface, which returns a choice without generating text.
• Sophtron 9B (Q8) ran on our own AWS g6e.xlarge with one L40S, one request at a time, through an SSH tunnel.
• Calls within a model were sequential. There were no retries and no best-of-n.

How cost is counted

OpenRouter models: the billed amount OpenRouter returned for each call.

Sophtron: $1.861 per hour divided by 3,600, times the measured seconds of that call, then divided by concurrent request count of 15. The assumption is one L40S with 48GB VRAM running the 9B model at Q8 (as benchmarked) with vLLM can handle 15 concurrent requests. During the benchmark, Sophtron's requests used 1,000–3,000 input tokens and 0–500 output tokens. Latency was measured one request at a time; with 15 concurrent requests per-call time would likely rise, so this cost is an estimate, not a measurement.

Logged in account decisions

We also hold 391 decisions recorded on logged in account pages. These are real accounts Sophtron owns for testing. They hold no customer data, but the pages still show real account details, so the outside models received only a redacted copy, which replaces names, account numbers, amounts, contact details and session tokens with consistent stand-ins and keeps the institution names. Before sending it we checked that redaction does not change the task: Sophtron chose the same option on 384 of 391 redacted cases as on the originals, with 372 and 371 correct. The redacted set ran once per model, and every OpenRouter request was restricted to providers that keep no data (Amazon Bedrock for Opus, Google for Sonnet, Azure for GPT, DeepInfra for Qwen, TypeSafe for Jev).

These providers differ from the ones that served Claude and GPT on the public sets, so their times are not directly comparable across the two sections. Logged in account pages are longer than the public ones, which raises time and cost for every model.

All three sets together

The 775 decisions of all three sets together: 204 historical, 180 from new institutions and 391 redacted logged in account decisions. Every model answered all of them.

• Smaller models cost 1/130 to 1/290 of Opus. Jev costs 1/130 of Opus over all 775 decisions; Sophtron on one rented L40S costs 1/290.
• Smaller models take 1/2 to 1/3 of the time. Jev and Sophtron answer 2.3× and 2.5× faster than Opus on average.
• Accuracy is close, not equal. Over all sets our models are 1.3 and 1.8 points behind Opus. They tie it on the public sets; the gap is on logged in account pages.
• Hosted Qwen 27B is as accurate as Opus but slow, 38 s per call on average and up to 197 s at P95 on logged in account pages.
• Sonnet's gap is output format. 147 of its 775 replies put an explanation before the JSON; counting those, it scores 94.8%.
• The runs were clean. 2 network errors in 4,650 calls. Each case ran once, so differences of one or two points are within noise.

Accuracy by set

Our models against Opus

Bold marks the best score in each set. Cost over all sets is the mean over every call that reported a charge, so the longer, pricier logged in account pages weigh in by their share of calls.

What this does not show

• End-to-end success. These are single decisions on frozen pages. Signing in, navigation, waiting for pages and downloads are not part of the measurement.
• Ground truth. Accepted answers were labelled during development with help from an LLM assistant, not by independent blind human review.
• Every site. The 130 cases are a selected sample from 42 sites, not a random draw.
• The format rule. Claude Sonnet 5.5 often wrote a short explanation before its JSON (60 calls). Strict scoring gives it 79.2%; accepting those answers gives 92.2%. Both are shown.

Published October 2026. Every figure in this doc is computed from the saved run records.