Essay

The Variability Problem

Same model. Same warehouse. Ten runs. Different answers.

VuonJune 20267 min read

If you run a production AI data agent, you have already seen this. Ask the same analytical question ten times and you will not get the same answer every time. Sometimes the numbers drift by a percentage point. Sometimes the methodology flips. Sometimes the agent quietly re-defines the population and answers a different question.

This is the variability problem in AI data analysis. It is invisible from a single demo. It is the gap between an answer that looks right and an answer that reproduces.

The interesting part is that the variance has structure. The same question, run repeatedly, fails in three structurally distinct ways, at three different layers of the SQL the agent writes. We call them population drift, methodology drift, and implementation drift. Each is invisible from the agent's output. Each requires a different fix.

Three archetypes of variability

The same agent giving different answers across runs is not random noise. The variance lives at three distinct layers, each with a different signature and a different remedy.

Population drift

The agent quietly answers for a different population.

Methodology drift

The agent picks a different method for the same question.

Implementation drift

Same method, same tables; one line of SQL differs.

Archetype 1 · Population drift

The agent quietly answers for a different population.

Most analytical questions assume a population: all users, every customer, every account. That population is rarely a single table in the warehouse. The agent has to construct it: pick the right entity table, decide which filters to keep, decide whether to include inactive or soft-deleted rows. When it skips that construction step and anchors on whichever table contains the metric being measured, the population silently collapses to the rows that already produced the metric.

The math inside that collapsed population is fine. The conclusion is wrong because the agent is now answering a different question than the one it was asked. The output still reads as a fluent analysis. The methodology section talks about "users" or "customers" without specifying which definition was used. From any summary above the SQL, the population swap is invisible.

In our warehouse, you ask for the average number of dashboards per organization. Right anchor starts from dim_organizations and counts zero-dashboard orgs as zero: mean 5.41. Wrong anchor groups from app_dashboards, which silently collapses the population to the 14,867 organizations that already have a dashboard: mean 6.03. Eight of ten trials picked the wrong anchor. Both numbers are real. Only one answers the question.

Right anchor · all organizations

SELECT
  o.org_id,
  COUNT(d.dashboard_id) AS dashboards
FROM dim_organizations o
LEFT JOIN app_dashboards d
  ON d.org_id = o.org_id
  AND d.deleted_at IS NULL
GROUP BY 1
-- AVG(dashboards) = 5.41

Returns all orgs (zero-dashboard included). Mean = 5.41.

Wrong anchor · orgs with dashboards

SELECT
  d.org_id,
  COUNT(*) AS dashboards
FROM app_dashboards d
WHERE d.deleted_at IS NULL
GROUP BY 1
-- AVG(dashboards) = 6.03

Returns 14,867 rows. Mean = 6.03.

Archetype 2 · Methodology drift

The agent picks a different method for the same question.

Methodology drift is path selection. A well-formed question often admits more than one reasonable computational path: median versus mean per DAU, a funnel measured by sessions or by users, retention against a fixed cohort or a rolling baseline. The paths produce different answers. The agent does not commit to one and stay there. Run to run, it picks one path, then the other.

The variance is not noise around a true value. It is the gap between two true-looking values. Both answers come with sensible prose, matching units, and the same-shape supporting tables. The methodology summary often describes the family of approaches without disambiguating which one was chosen. The clue is buried in the GROUP BY or the join key.

In our warehouse, you ask for the median sessions per DAU by industry in Q4. Method A medians over user-day rows: 1 per industry. Method B medians over industry-day ratios: 1.43 per industry. In ten runs the agent picked Method B eight times and Method A twice. Whichever method got picked, the answer pegged to that method exactly.

Method A · per user-day rows

WITH user_day AS (
  SELECT
    user_id, industry,
    COUNT(*) AS sessions
  FROM events_session_start
  WHERE quarter = 'Q4'
  GROUP BY 1, 2
)
SELECT industry,
  PERCENTILE_CONT(0.5)
    WITHIN GROUP (ORDER BY sessions)
    AS median_sessions
FROM user_day
GROUP BY 1

median_sessions = 1 per industry.

Method B · per industry-day ratios

WITH industry_day AS (
  SELECT industry,
    COUNT(*) AS sessions,
    COUNT(DISTINCT user_id) AS dau
  FROM events_session_start
  WHERE quarter = 'Q4'
  GROUP BY 1
)
SELECT industry,
  PERCENTILE_CONT(0.5)
    WITHIN GROUP (
      ORDER BY sessions::numeric / dau
    ) AS median_ratio
FROM industry_day
GROUP BY 1

median_ratio = 1.43 per industry.

Archetype 3 · Implementation drift

Same method, same tables; one line of SQL differs.

Implementation drift is what is left after population and methodology are locked in. The agent has agreed on what to measure and how to measure it. But SQL is verbose, and "the same method" still leaves dozens of small choices: which boolean predicates belong in the filter, how to coalesce nulls, whether to exclude internal accounts, whether to treat soft-deletes as active. None of these feel methodologically significant. Each one shifts the number by a fraction of a point.

The signature is that the methodology rationale is identical across runs. The query plan is nearly identical. The directional story holds. Only the absolute numbers wobble. The cumulative drift looks like noise around a true value, but it is not noise. It is unsurfaced choices. From any summary above the SQL, the trials are indistinguishable.

In our warehouse, you ask for the quarter-over-quarter abandonment rate. Same shape, same tables, same quarters across every run. One run includes AND duration_seconds > 60 in the cancelled filter; another omits it. Q3 abandoned drops from 6,013 to 5,813. Rate moves from 37.7% to 36.9%. The directional finding (Q3-to-Q4 improvement) holds. The absolute rate is off by a point.

Run A · no filter

SELECT
  DATE_TRUNC('quarter', flow_started_at)
    AS quarter,
  COUNT(*) FILTER (
    WHERE flow_status = 'cancelled'
  ) AS abandoned,
  COUNT(*) AS total_flows
FROM flows
WHERE flow_started_at >= '2025-04-01'
GROUP BY 1

Q3 abandoned = 6,013. Rate = 37.7%.

Run B · with sub-minute filter

SELECT
  DATE_TRUNC('quarter', flow_started_at)
    AS quarter,
  COUNT(*) FILTER (
    WHERE flow_status = 'cancelled'
    AND duration_seconds > 60
  ) AS abandoned,
  COUNT(*) AS total_flows
FROM flows
WHERE flow_started_at >= '2025-04-01'
GROUP BY 1

Q3 abandoned = 5,813. Rate = 36.9%.

Why this matters

Your agent in production answers the same question more than once. One answer lands in the Board deck. Another lands in a different team's dashboard. When those disagree, you have a major credibility problem.

An agent that's wrong reliably can be spotted and fixed. But one that's right unreliably goes unnoticed, quietly eroding trust.

The metric for this is pass^k, proposed in the τ-bench paper and adopted across the agent-quality literature. If a task is run k independent times, what fraction of the time does the agent get the right answer every time? pass^1 is standard single-attempt accuracy. pass^k for higher k is the reliability decay.

Here is what the decay looks like across a ten-question subset of our benchmark from the previous post. Our production agent inside the Vuon harness, versus the same Claude model with the data catalog in its prompt. Ten independent trials per question.

Reliability decay across ten runs (pass^k)

k = 1
66%
58%
k = 2
55%
47%
k = 4
47%
37%
k = 8
42%
30%
k = 10
40%
30%
VuonClaude + catalog

Ten reviewed analytics questions drawn from our benchmark in the previous post, ten independent trials per question. pass^k is the chance all k attempts produce the right answer; pass^1 is standard single-attempt accuracy.

Vuon's harness shows an 8-point edge on pass^1 and widens to 12 points by pass^4. Both configurations are capable enough to answer the majority of these questions correctly once. The 12-point gap is variability. Same model, same warehouse, same question, different SQL on different runs. The harness narrows it by constraining what the agent can do at each layer.

What reduces variability

No single intervention removes variability. The Vuon harness reduces it by constraining each of the three layers separately.

Contextually aware SQL compilation

PopulationMethodology
  1. The semantic graph tags each fact with its natural grain and the entity it belongs to: sessions are user-day facts; dashboards belong to organizations.
  2. The compiler traces both through the agent's SQL, following CTEs and joins to track what each intermediate result represents.
  3. At the aggregation step, it compares what the agent is operating on against what the question requires. A mismatch is rejected before execution.

Semantic and policy graph

PopulationMethodology
  1. Every entity, metric, and computation in your warehouse has a canonical definition stored in the harness's semantic graph: what "users" means, how "median by industry" is computed, what tables to join for share-of questions.
  2. These definitions are loaded as constraints when the agent writes SQL, not stuffed into a prompt as suggestions.
  3. The agent's query has to satisfy the definitions; if it doesn't, the conflict is detected and resolved before execution.

Post-execution validation

Implementation
  1. Every governed metric has a baseline stored: the expected answer, the expected range, sometimes the expected distribution shape, derived from a trusted version of the query.
  2. After the agent's SQL runs, the validator compares the result against the baseline.
  3. If the result drifts beyond a threshold (a point on a stable metric, an outlier in a distribution that should be smooth), the run is flagged and the SQL diff is surfaced for review.

Tracked and versioned calculations

All three (visibility)
  1. Every calculation the agent produces is stored as a versioned record: the SQL it ran, the inputs it used, the definitions it bound against, the result it returned.
  2. Two runs of the same question produce two records.
  3. The records can be diffed at every level (SQL text, query structure, semantic definitions used, numerical results), so you can see exactly what changed between runs.

Bottom line

Variability in AI data agents is structural. Three drift patterns, three layers of the SQL. Closing the gap requires both a metric and a mechanism: pass^k to measure reliability, and a harness that constrains the agent at each layer.

If you are shipping a production AI data agent without both, you can claim accuracy. You cannot claim reliability.