One Developer Benchmarked 20 OpenRouter Providers on the Same Model and Found a 20-Point Quality Swing
Mo Moustafa ran rolling 32-day benchmarks across more than 20 OpenRouter providers serving identical DeepSeek V4 Flash weights and found GPQA scores ranging from 75% to 90%. Same model, same weights, 20-point gap. The post hit 698 points on Hacker News.
The WJS Desk
Sep 12, 2026 · updated 12 days ago · 5 min read
Mo Moustafa spent 32 days running rolling benchmarks across more than 20 providers on OpenRouter, all serving the same DeepSeek V4 Flash 0731 model weights. We read the full post, now sitting at 698 points on Hacker News, and verified the methodology. The results show quality swings of 15 to 30 percentage points on identical weights depending on which provider serves the request. First-party DeepSeek scored 90% on GPQA knowledge tests. DigitalOcean scored 75%. That 20-point gap is not noise.
We have used OpenRouter in our own tooling, so we read this with more than casual interest. The post documents ten distinct failure modes and arrives at a conclusion that should worry anyone building on model routing: the provider, not the model, is the variable that determines your output quality.
The benchmark methodology
Moustafa ran GPQA (graduate-level knowledge questions) and TAU-Bench (tool-augmented understanding) across the provider pool as rolling 32-day averages. The same prompt, the same model name, the same OpenRouter API call, different providers handling the backend. The results are reported as scatter plots showing each provider's score, and the spread is wide enough that picking the wrong provider is equivalent to downgrading to a weaker model.
| Provider | GPQA Score | Notes |
|---|---|---|
| DeepSeek (first-party) | 90% | Baseline, running own weights |
| DigitalOcean | 75% | 20-point gap on identical model |
The tool-calling benchmark (TAU-Bench) showed similar spreads. The takeaway from Moustafa's data: "a 20 point swing is not noise" for tool-calling tasks, and the variance is not explained by declared quantisation levels. Providers declaring fp4 precision were scattered throughout the fp8 performance range, leading Moustafa to conclude that "precision is a bad proxy for quality."
Ten failure modes, documented
The post catalogues problems that go beyond performance variance:
Vision model blindness. DeepInfra's Qwen3.5 misread the letter K as R and called red blue on image recognition tasks. Venice and Together did not process MiniMax images at all. Other providers running the same model handled the same images correctly.
Reasoning effort silently ignored. The reasoning.effort parameter is accepted by every provider (no errors returned) but does not consistently function. DigitalOcean, GMICloud, Mancer, and Venice showed minimal response to effort level changes. You send the parameter, get a 200 back, and the model's behaviour does not change.
Empty successful responses. Providers occasionally returned HTTP 200 with null content and null reasoning fields. The API call succeeded. The response contained nothing usable.
Tool call parsing failures. Some providers returned raw XML markup in responses instead of structured tool call data. The model generated the tool call correctly; the provider's parsing layer mangled it.
Missing usage data. Some endpoints returned no usage object at all, making it impossible to track token consumption or verify that reasoning tokens were actually generated.
Inconsistent history requirements. Different providers enforce different constraints on how reasoning content appears in conversation history. The API contract is not per-model; it is per-provider. Code that works on one backend breaks on another serving the same model.
Geolocation rate limiting. Venice and Novita worked from Moustafa's development Mac but returned 429 errors from production infrastructure. Testing locally does not predict production behaviour.
Provider reliability collapse. Moustafa tried pinning three "reliable" providers as fallbacks. Within the testing period, Baidu rate-limited, Cloudflare stopped serving the model entirely, and Alibaba rate-limited everything. Three fallbacks, all three failed.
The recommendations
Moustafa's practical advice is specific enough to be immediately useful:
- Monitor per-provider benchmarks on your actual workload, not the model's published scores.
- Parse tool calls on your end to handle provider parsing failures.
- Test from production infrastructure, not development machines.
- Do not pin single providers. Use fallback routing, but expect fallbacks to fail too.
- Track reasoning tokens per provider to verify that the effort parameter actually changes behaviour.
- Treat HTTP 200 as provisional. Validate actual content before accepting responses.
The trust problem: When a provider accepts a parameter, returns a success code, and silently ignores what you asked for, there is no way to detect the failure without running your own benchmarks. OpenRouter surfaces provider availability and pricing. It does not surface provider quality, parsing correctness, or parameter compliance.
What OpenRouter's architecture makes hard
OpenRouter exists to abstract away provider choice. You pick a model, OpenRouter picks a provider (or lets you pin one), and the routing happens behind a single API endpoint. This is valuable for discovery and fallback. It is also the source of the problem Moustafa documents.
When the routing layer does not account for quality variance, it can send your request to the cheapest or fastest provider, which may also be the one with a 20-point GPQA deficit. The model name gives you a false sense of determinism. You think you are calling DeepSeek V4 Flash. You are calling whichever DeepSeek V4 Flash happens to be available, quantised however that provider chose, parsed by whatever middleware they run, subject to whatever rate limits their infrastructure imposes.
This is not an OpenRouter-specific problem, and we want to be clear about that. Any multi-provider routing layer (including self-built ones) has the same issue. OpenRouter just happens to be the one that 698 Hacker News voters use.
Whether this matters
If you are building a product on OpenRouter, this post is required reading. A 20-point quality swing on identical weights means your application's output quality is partially determined by infrastructure decisions you did not make and cannot observe. For tool calling, where a parsing failure means your agent takes the wrong action, the consequences are worse than a quality dip.
If you are using OpenRouter for personal projects or experimentation, the variance probably does not matter. You notice when the output is bad and retry.
The model name on your API call is not the thing that determines your output quality. The provider is, and nobody benchmarks providers.
What would change the picture: OpenRouter publishing per-provider quality metrics alongside pricing, or exposing a quality routing preference alongside cost and latency. Until then, the due diligence Moustafa did is the due diligence we think every production user needs to do themselves, and most are not.
