Benchmark
The Harness Premium:
A 20-Question Benchmark
Same Claude model. Same warehouse. 3 configurations. 20 points of spread.
The conversation has shifted from which model to what context. That's right: a strong semantic layer—metric definitions, a clean dimensional model, governance—is necessary infrastructure for any AI agent that produces correct numbers on enterprise data.
But context is two things. The catalog is one. What makes the agent actually use the catalog, instead of writing around it, is the other. We tested them apart. 3 configurations of the same Claude model, against 20 reviewed analytics questions on a real customer warehouse:
- Raw schema access answered 45% correctly.
- Adding a governed metric catalog and dimensional model to the prompt brought it to 50%.
- The same model running inside Vuon's harness answered 65%.
The catalog earned a quarter of the lift. The harness earned three-quarters. Below: why the harness wins at the mechanism level, what the other 3 judges show, and how the gap shows up across all 20 questions.
How we tested
We ran 20 analytics questions against a benchmark warehouse modeled on a real product-analytics platform. Each question has an adjudicated correct answer reviewed by the team that owns the data. Configurations were scored on a single attempt (pass^1). The set ranges from straightforward to multi-step analytical questions involving cohorts, distributions, and retention.
The 3 configurations all ran the same Claude model against the same BigQuery warehouse.
- Claude + schema. The model with raw warehouse access. No metric definitions or dimensional model surfaced.
- Claude + catalog. The same model with Vuon's governed metric definitions and dimensional model loaded into its context window as structured reference material.
- Vuon. The same model running inside Vuon's production harness: discovery, SQL compilation, semantic graph, post-execution validation, versioned artifact store.
We graded each answer against 4 judges.
Correctness
Did the agent reach the adjudicated answer without a materially false claim?
Methodology
Was the approach sound relative to the expected analytical approach?
Completeness
Did the answer cover the components the question asked for?
Discovery
Did the agent find the right tables and metrics?
What we found
Correctness is the headline number. Methodology, completeness, and discovery tell you why correctness is what it is. Reading them together—alongside the mechanism table and the cases—is where the story lives.
The correctness ladder
Correctness across 20 reviewed analytics questions
Same frontier model and warehouse across all three. Single-turn (pass^1): one attempt per question, no retries. n=20 reviewed analytics questions. A larger run and reliability metrics (pass^k) are coming in follow-up posts.
Adding the catalog and dimensional model to a frontier model's prompt buys 5 points of correctness against a raw-schema baseline. Wrapping the same model in Vuon's harness buys 15 more. That harness has three pieces: a discovery layer that decides which metric to invoke, a SQL compilation step that verifies invocations against the semantic graph, and a post-execution validation pass that catches outputs that don't match it.
How the harness wins
The 15-point gap to Vuon comes from four mechanisms in the harness. The Technical Overview describes them as architecture; the cases below describe what they look like when they fire.
Contextually aware SQL compilation
A compilation layer that understands warehouse structure and business definitions. It flags queries that contradict them, in the same spirit that a code compiler flags undefined variables or invalid types.
Where it showed up · Case 1
"How concentrated is dashboard creation among users?"
The catalog's definition of users includes everyone, not just users who already do the thing being measured. Vuon's compilation step kept the analysis anchored to the full user population, surfacing the 80.8% who created zero dashboards. Claude + catalog took the natural shortcut and analyzed only the users who had created something, landing on "moderately concentrated." Same data, two different denominators, opposite conclusion.
The judge
"The agent only analyzed users who created at least one dashboard, completely ignoring the 80%+ of users who never created any dashboards. A fundamental mischaracterization."
Will a better model close this? The catalog gets the agent to a sound idea. What turns that idea into a correct number is what runs on top of it.
The other judges
Methodology shows the biggest catalog lift (+0.15) and the smallest harness lift (+0.05). It rewards an agent for reasoning toward a sound approach, even when the SQL departs from it, so the catalog earns partial credit just for surfacing structured material to reason about. The harness's smaller methodology lift reflects where it does its work instead: closing the execution gap in correctness.
Discovery and Completeness are the negative-space judges. Discovery is the tightest band: all 3 configurations have BigQuery access, so finding the right tables isn't where the variance lives. Completeness sits near the ceiling (0.93–1.00) for all 3, which is the warning itself. Every configuration produces complete-looking answers; only one produces consistently correct ones.
The full scoreboard
Every question we tested, grouped by outcome. The 4 wins concentrate where the harness's scoping caught a divergence the model alone missed. The 1 loss is a grain ambiguity the harness's reasoning doesn't yet disambiguate. The 6 cases where everyone failed cluster on causal discipline and denominator scoping: limits we've measured, named, and aren't claiming to have solved.
How concentrated is dashboard creation among users?
Vuon anchored to all users (including the 80.8% who created nothing); Claude analyzed only users who had created at least one dashboard.
How does engagement differ between paid and free-tier organizations?
Vuon scoped "paid" to active subscriptions and normalized to active days; Claude used raw cumulative counts and lumped churned orgs in with paid.
Compare dashboard creation abandonment rate Q4 vs Q3 2025.
Vuon used the rubric's canonical scope for assigning flows to Q3 vs Q4; Claude's scope diverged slightly and the counts didn't reconcile.
What fraction of last year's sessions came from the top 10% of users?
Vuon probed the data and read "last year" as 2025 (39.1%); Claude defaulted to 2024 (43.4%) and produced a materially wrong number.
Bottom line
The model isn't the variable. The system around it is. Across 3 configurations of the same Claude model on 20 reviewed analytics questions, the catalog earned a quarter of the 20-point lift; the harness earned the rest. We'll keep testing. We'll keep publishing.